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1.  Introduction 


Adaptive  training  and  educational  instruction  provides  tailored  experiences  that  are 
guided  by  computer-based  intelligent  tutoring  systems.  This  report  provides  a 
comprehensive  bibliography  of  publications  produced  from  2016  through  2017 
under  the  direction  of  the  US  Army  Research  Laboratory’s  (ARL’s)  adaptive 
training  and  education  research  program.  The  bibliography  includes  references  for 
journal  articles,  technical  reports,  and  conference  papers  produced  by  ARL 
employees  and  contractors  to  support  the  development  of  the  Generalized 
Intelligent  Framework  for  Tutoring  (GIFT)  prototypes  during  this  period  to  provide 
an  open-source  architecture  for  authoring,  delivering,  and  evaluating  adaptive 
instruction.  In  addition  to  the  GIFT  symposia,  conference  papers  were  produced 
and  presented  at  a  variety  of  technical  venues  that  included,  but  were  not  limited  to, 
the  following  organizations’  conferences:  Interservice/Industry  Training, 
Simulation  and  Education,  Artificial  Intelligence  in  Education,  Intelligent  Tutoring 
Systems,  Human  Eactors  and  Ergonomics,  Applied  Human  Eactors  and 
Ergonomics,  Augmented  Cognition,  Defense  and  Homeland  Security  Simulation, 
and  Elorida  Artificial  Intelligence  Research  Society.  Contributors  to  these 
publications  included  members  of  US  government  agencies,  academic  institutions 
worldwide,  and  industry  in  the  modeling,  simulation,  and  training  industrial  base. 
References  are  arranged  in  reverse  chronological  order  grouped  by  their  year  of 
publication. 

2.  Bibliography 
2017  Publications 

Special  Issue  on  the  Generalized  Intelligent  Framework  for 

Tutoring  (GIFT):  Creating  a  Stable  and  Flexible  Platform  for 
Innovations  in  AIED  Research 

Abstract:  The  goal  of  this  research  was  the  development  of  a  practical  architecture 
for  the  computer-based  tutoring  of  teams.  This  article  examines  the  relationship  of 
team  behaviors  as  antecedents  to  successful  team  performance  and  learning  during 
adaptive  instruction  guided  by  Intelligent  Tutoring  Systems  (ITSs).  Adaptive 
instruction  is  a  training  or  educational  experience  tailored  by  artificially-intelligent, 
computer-based  tutors  with  the  goal  of  optimizing  learner  outcomes  (e.g., 
knowledge  and  skill  acquisition,  performance,  enhanced  retention,  accelerated 
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environments).  The  core  contribution  of  this  research  was  the  identification  of 
behavioral  markers  associated  with  the  antecedents  of  team  performance  and 
learning  thus  enabling  the  development  and  refinement  of  teamwork  models  in  ITS 
architectures.  Teamwork  focuses  on  the  coordination,  cooperation,  and 
communication  among  individuals  to  achieve  a  shared  goal.  For  ITSs  to  optimally 
tailor  team  instruction,  tutors  must  have  key  insights  about  both  the  team  and  the 
learners  on  that  team.  To  aid  the  modeling  of  teams,  we  examined  the  literature  to 
evaluate  the  relationship  of  teamwork  behaviors  (e.g.,  communication,  cooperation, 
coordination,  cognition,  leadership/coaching,  and  conflict)  with  team  outcomes 
(learning,  performance,  satisfaction,  and  viability)  as  part  of  a  large-scale  meta¬ 
analysis  of  the  ITS,  team  training,  and  team  performance  literature.  While  ITSs 
have  been  used  infrequently  to  instruct  teams,  the  goal  of  this  meta-analysis  make 
team  tutoring  more  ubiquitous  by:  identifying  significant  relationships  between 
team  behaviors  and  effective  performance  and  learning  outcomes;  developing 
instructional  guidelines  for  team  tutoring  based  on  these  relationships;  and  applying 
these  team  tutoring  guidelines  to  the  Generalized  Intelligent  Framework  for 
Tutoring  (GIFT),  an  open  source  architecture  for  authoring,  delivering,  managing, 
and  evaluating  adaptive  instructional  tools  and  methods.  In  doing  this,  we  have 
designed  a  domain-independent  framework  for  the  adaptive  instruction  of  teams. 

Keywords:  Team  tutoring.  Teamwork.  Taskwork.  Adaptive  instruction.  Team- 
based  instruction.  Team  learning.  Team  performance.  Team  satisfaction.  Team 
viability.  Collective.  Collective  training.  Collaborative  learning. 

Citation:  Sottilare,  R.A.,  Graesser,  A.C.,  Lester,  J.,  &  Baker,  R.  (2017).  Special 
Issue  on  the  Generalized  Intelligent  Framework  for  Tutoring  (GIFT):  Creating  a 
Stable  and  Flexible  Platform  for  Innovations  in  AIED  Research.  International 
Journal  of  Artificial  Intelligence  in  Education.  DOT  10.1007/s40593-017-0149-9. 

Reusing  Evidence  in  Assessment  and  Intelligent  Tutors 

Abstract:  Assessment  design  methodologies  such  as  evidence-centered  design 
(ECD)  provide  an  approach  for  representing  the  “argument”  underlying  an 
assessment.  Argument-based  structures  articulate  the  chain  of  reasoning  connecting 
task-level  data  to  the  evidence  required  to  support  assessment  claims  about  the 
student.  Intelligent  tutoring  systems  (ITSs)  reflect  similar  underlying  rationale, 
with  decisions  about  how  to  update  the  student  model  and  what  to  present  to  the 
student  next,  dependent  on  student  performance  (evidence).  Although  the  concept 
of  “evidence”  is  widely  used  within  the  assessment  community,  it  is  less  well 
articulated  in  the  ITS  literature.  In  this  chapter,  we  argue  for  the  necessity  of 
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facilitate  the  use  of  evidence  in  ITSs.  Explicitly  representing  evidence  may  provide 
a  stronger  theoretical  basis  for  an  ITS  and  facilitate  the  reuse  of  assessment 
components  across  ITSs.  Furthermore,  additions  to  the  Generalized  Intelligent 
Framework  for  Tutoring  (GIFT)  may  support  defining  links  from  observable 
evidence  to  claims  (number  of  links,  types  of  evidence,  types  of  observables,  and 
properties)  as  well  as  mechanisms  for  quantifying  evidence.  Several  examples 
illustrate  how  this  approach  facilitates  reuse  and  handling  of  evidence  in  the  areas 
of  conversation-based  assessment  and  GIFT-medical  prototypes. 

Keywords:  Intelligent  tutors.  Concept  of  evidence.  Generalized  Intelligent 
Framework  for  Tutoring. 

Citation:  Zapata-Rivera,  D.,  Brawner,  K.,  Jackson,  G.  T.,  &  Katz,  I.  R.  -Reusing 
Evidence  in  Assessment  and  Intelligent  Tutors.  Design  Recommendations  for 
Intelligent  Tutoring  Systems,  125. 

Affect  Dynamics  in  Military  Trainees  using  vMedic:  From 
Engaged  Concentration  to  Boredom  to  Confusion 

Abstract:  The  role  of  affect  in  learning  has  received  increasing  attention  from 
AIED  researchers  seeking  to  understand  how  emotion  and  cognition  interact  in 
learning  contexts.  The  dynamics  of  affect  over  time  have  been  explored  in  a  variety 
of  research  environments,  allowing  researchers  to  determine  the  extent  to  which 
common  patterns  are  captured  by  hypothesized  models.  This  paper  present  an 
analysis  of  affect  dynamics  among  learners  using  vMedic,  which  teaches  combat 
medicine  protocols  as  part  of  the  military  training  at  West  Point,  the  United  States 
Military  Academy.  In  doing  so,  we  seek  both  to  broaden  the  variety  of  learning 
contexts  being  explored  in  order  better  understand  differences  in  these  patterns  and 
to  test  the  theoretical  predictions  on  the  development  of  affect  over  time. 

Keywords:  Affect  dynamics.  Emotion  and  cognition  interaction.  Military  training. 

Citation:  Ocumpaugh,  J.,  Andres,  J.  M.,  Baker,  R.,  DeFalco,  J.,  Paquette,  F.,  Rowe, 
J.,  ...  &  Sottilare,  R.  (2017,  June).  Affect  Dynamics  in  Military  Trainees  using 
vMedic:  From  Engaged  Concentration  to  Boredom  to  Confusion.  In  International 
Conference  on  Artificial  Intelligence  in  Education  (pp.  238-249).  Springer,  Cham. 
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Generalized  Intelligent  Framework  for  Tutoring  (GIFT) 

Cloud/Virtual  Open  Campus  Quick  Start  Guide  (Revision  1) 

Abstract:  This  document  serves  as  the  quick-start  guide  for  GIFT  Cloud,  the  web- 
based  application  version  of  the  Generalized  Intelligent  Framework  for  Tutoring 
(GIFT).  GIFT  is  a  modular,  open-source  framework  for  building,  deploying,  and 
managing  adaptive  training  content.  GIFT  Cloud  allows  learners,  authors,  and 
researchers  to  access  GIFT  without  downloading  and  installing  software  on  a 
desktop  or  laptop  computer.  The  purpose  of  this  document  is  to  provide  an 
overview  of  the  primary  interfaces  and  basic  functions  of  GIFT  Cloud.  Topics 
include  establishing  a  GIFT  Account,  taking  courses  on  GIFT  Cloud,  orienting  the 
user  to  GIFTS  authoring  tools,  and  locating  resources  and  providing  feedback  on 
GIFT  Cloud.  This  document  serves  as  a  revision  to  ARL-CR-0796,  to  reflect  recent 
developments  to  GIFT  interfaces,  processes,  and  terminology. 

Keywords:  Generalized  Intelligent  Framework  for  Tutoring.  Virtual  open  campus. 
GIFT  cloud.  GIFT  authoring  tools. 

Citation:  Ososky,  S.  (2017).  Generalized  Intelligent  Framework  for  Tutoring 
(GIFT)  Cloud/Virtual  Open  Campus  Quick  Start  Guide  (Revision  1)  (No.  ARL- 
CR-0816).  Army  Research  Lab,  Orlando,  FL. 

Designing  Adaptive  Instruction  for  Teams:  A  Meta-analysis 

Abstract:  The  goal  of  this  research  was  the  development  of  a  practical  architecture 
for  the  computer-based  tutoring  of  teams.  This  article  examines  the  relationship  of 
team  behaviors  as  antecedents  to  successful  team  performance  and  learning  during 
adaptive  instruction  guided  by  Intelligent  Tutoring  Systems  (ITSs).  Adaptive 
instruction  is  a  training  or  educational  experience  tailored  by  artificially-intelligent, 
computer-based  tutors  with  the  goal  of  optimizing  learner  outcomes  (e.g., 
knowledge  and  skill  acquisition,  performance,  enhanced  retention,  accelerated 
learning,  or  transfer  of  skills  from  instructional  environments  to  work 
environments).  The  core  contribution  of  this  research  was  the  identification  of 
behavioral  markers  associated  with  the  antecedents  of  team  performance  and 
learning  thus  enabling  the  development  and  refinement  of  teamwork  models  in  ITS 
architectures.  Teamwork  focuses  on  the  coordination,  cooperation,  and 
communication  among  individuals  to  achieve  a  shared  goal.  For  ITSs  to  optimally 
tailor  team  instruction,  tutors  must  have  key  insights  about  both  the  team  and  the 
learners  on  that  team.  To  aid  the  modeling  of  teams,  we  examined  the  literature  to 
evaluate  the  relationship  of  teamwork  behaviors  (e.g.,  communication,  cooperation. 
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coordination,  cognition,  leadership/coaching,  and  conflict)  with  team  outcomes 
(learning,  performance,  satisfaction,  and  viability)  as  part  of  a  large-scale  meta¬ 
analysis  of  the  ITS,  team  training,  and  team  performanee  literature.  While  TTSs 
have  been  used  infrequently  to  instruet  teams,  the  goal  of  this  meta-analysis  make 
team  tutoring  more  ubiquitous  by:  identifying  significant  relationships  between 
team  behaviors  and  effeetive  performanee  and  learning  outeomes;  developing 
instruetional  guidelines  for  team  tutoring  based  on  these  relationships;  and  applying 
these  team  tutoring  guidelines  to  the  Generalized  Intelligent  Framework  for 
Tutoring  (GIFT),  an  open  souree  arehiteeture  for  authoring,  delivering,  managing, 
and  evaluating  adaptive  instructional  tools  and  methods.  In  doing  this,  we  have 
designed  a  domain-independent  framework  for  the  adaptive  instruction  of  teams. 

Keywords:  Team  tutoring.  Teamwork.  Taskwork.  Adaptive  instruction.  Team- 
based  instruetion.  Team  learning.  Team  performanee.  Team  satisfaetion.  Team 
viability.  Colleetive.  Colleetive  training.  Collaborative  learning. 

Citation:  Sottilare,  R.  A.,  Burke,  C.  S.,  Salas,  E.,  Sinatra,  A.  M.,  Johnston,  J.  H., 

&  Gilbert,  S.  B.  (2017).  Designing  adaptive  instruction  for  teams:  A  meta-analysis. 
International  Journal  of  Artificial  Intelligence  in  Education,  1-40. 

Shared  Mental  Models  in  Support  of  Adaptive  Instruction  for 
Teams  Using  the  GIFT  Tutoring  Architecture 

Abstract:  Teams  and  teamwork  are  ubiquitous  in  military  and  civilian 
organizations.  Their  importanee  to  organizational  sueeess  eannot  be  overstated. 
This  article  deseribes  the  relationship  and  effect  of  three  concepts:  Intelligent 
Tutoring  Systems  (ITSs),  shared  mental  models,  and  teamwork.  The  nexus  between 
these  eoneepts  is  examined  to  determine  its  capability  to  support  adaptive 
instruction  of  teams,  defined  here  as  collectives  of  interdependent  individuals  who 
must  eommunicate  and  interact  with  each  other  in  order  to  perform  assigned  tasks 
and  missions.  An  assumption  underlying  this  examination  is  that  augmenting  the 
mental  modeling  processes  of  ITS  with  the  mental  models  shared  by  members  of 
interdependent  teams  will  allow  the  considerable  and  increasingly  research- 
established  eapabilities  of  intelligent  tutoring  of  individuals  to  be  applied  in  training 
teams.  Specifically,  we  reviewed  the  learning  and  performance  literature  to  identify 
how  shared  mental  models  of  cognition  could  be  used  to  enhance  the  adaptive 
instruetion  of  teams.  Our  goal  is  to  develop  a  methodology  to  enhance  training  and 
educational  options  for  institutions  that  provide  adaptive  team  instruction  at  the 
point-of-need.  Toward  this  end,  we  discuss  the  adaptation  of  the  Generalized 
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Intelligent  Framework  for  Tutoring  (GIFT),  an  open  source  tutoring  architecture, 
to  accommodate  team  models  and  states.  While  this  article  makes  a  first  step  toward 
defining  a  process  for  team  tutoring,  challenges  remain.  Team  tutors  must  have  the 
ability  to  manage  uncertainty  and  the  dynamic  nature  of  team  interaction  and 
communication  in  order  to  make  effective  and  timely  decisions  that  optimize  team 
and  team  member  performance. 

Keywords:  Adaptive  instruction.  Shared  mental  models.  Collective  tasks. 
Cognitive  models.  Affective  models.  Social  models.  Team  tutoring.  Collective 
tutoring. 

Citation:  Fletcher,  J.  D.,  &  Sottilare,  R.  A.  (2017).  Shared  Mental  Models  in 
Support  of  Adaptive  Instruction  for  Teams  Using  the  GIFT  Tutoring  Architecture. 
International  Journal  of  Artificial  Intelligence  in  Education,  1-21. 

Enhancing  the  Experience  Application  Program  Interface  (xAPI) 
to  Improve  Domain  Competency  Modeling  for  Adaptive 
Instruction 

Abstract:  This  paper  describes  methods  for  enhancing  the  experience  application 
program  interface  (xAPI)  to  improve  the  assessment  of  domain  competency 
modeling  for  adaptive  instruction.  xAPI  is  an  e-leaming  software  specification 
which  allows  individual  learning  experiences  and  achievements  to  be  amassed  in  a 
Learning  Record  Store  (LRS).  Adaptive  instruction  includes  tailored  training  or 
educational  experiences  usually  delivered  and  guided  by  Intelligent  Tutoring 
Systems  (ITSs).  ITSs  can  more  effectively  tailor  or  adapt  instruction  when  they 
have  more  accurate  models  of  the  learner's  prior  knowledge  or  competency.  This 
paper  examines  the  potential  effect  of  methods  to  more  accurately  model  learner 
experiences  and  domain  competency  in  an  LRS.  Specifically,  we  recommend  five 
methods  to  improve  xAPI  statements  by  documenting:  1)  achievement  types;  2) 
experience  duration;  3)  experience  source  information;  4)  domain  learning  and 
forgetting;  and  5)  assessment  within  learning  experiences. 

Keywords:  Experience  application  program  interface.  Domain  competency 
modeling.  E-leaming  software  application.  Learning  Record  Store. 

Citation:  Sottilare,  R.  A.,  Long,  R.  A.,  &  Goldberg,  B.  S.  (2017,  April).  Enhancing 
the  Experience  Application  Program  Interface  (xAPI)  to  Improve  Domain 
Competency  Modeling  for  Adaptive  Instmction.  In  Proceedings  of  the  Eourth 
(2017)  ACM  Conference  on  Learning®  Scale  (pp.  265-268).  ACM. 
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Challenges  in  Authoring,  Instructional  Management,  and 
Evaluation  Methods  for  Adaptive  Instructional  Systems 

Abstract:  Many  Intelligent  Tutoring  Systems  (ITSs)  are  highly  effective  learning 
tools  and  provide  individual  adaptive  instruction  in  single,  well-defined,  cognitive 
task  domains  (e.g.,  mathematics,  physics,  or  software  programming).  The  adaptive 
instruction  provided  by  ITSs  intelligently  tailors  content,  adapts  the  curriculum  and 
guides  the  learner  with  the  goal  of  optimizing  learning.  In  his  meta- analysis, 
VanLehn  (2011)  notes  that  ITSs  have  evolved  to  parity  with  expert  human  tutors. 
Although  this  is  exciting  news,  ITSs  remain  expensive  to  author  based  on  their 
complexity,  their  lack  of  reusable  components,  and  the  expert  skillset  required  to 
design,  create  and  update  them.  While  it  may  be  impractical  at  this  time  to  develop 
ITSs  for  low  density  instructional  domains  (e.g.,  specialized  fields  with  small 
populations  of  learners),  the  large  number  of  potential  learners  in  high  density 
domains  (e.g.,  high  school  mathematics)  and  their  high  degree  of  effectiveness  help 
offset  their  initial  authoring  costs  and  demonstrate  their  potential.  What  if  ITSs 
were  easier  to  author  in  a  broader  range  of  task  domains?  What  if  their  return  on 
investment  (ROI)  made  the  authoring  of  even  low  density  adaptive  instructional 
domains  cost  effective? 

This  talk  will  focus  on  significant  challenges  and  emerging  solutions  related  to  the 
development  and  adoption  of  adaptive  instructional  systems  (AISs)  which  include: 
leamer(s);  ITSs  to  guide  learning  in  a  domain;  and  integrated  environments  (e.g., 
simulations,  playgrounds,  web  pages,  mobile  applications,  or  serious  games)  all  of 
which  interact  and  influence  each  other  with  the  goal  of  optimizing  learning.  To 
this  end  we  have  identified  five  challenges  or  barriers  to  the  practical  use  of  AISs: 

1)  reducing  the  time  and  skill  required  to  author  AISs;  2)  optimizing  adaptive 
instruction  of  individuals  and  teams  to  enhance  learning;  3)  building  rapport  and 
engagement  with  conversational  agents;  4)  supporting  adaptive  instruction  in  the 
classroom  and  distributed  learning  contexts,  and  5)  evaluating  the  true  effectiveness 
of  adaptive  instructional  tools  and  methods.  We  discuss  these  challenges  through 
the  lens  of  the  Generalized  Intelligent  Framework  for  Tutoring  (GIFT;  Sottilare, 
Brawner,  Goldberg  &  Holden,  2012),  an  adaptive  tutoring  prototype  architecture 
being  developed  with  the  goals  of  lowering  the  entry  skill  and  reducing  the  time 
required  to  author  adaptive  instruction,  automating  the  delivery  of  instruction,  and 
automating  the  evaluation  of  AISs,  components,  tools,  and  methods. 
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Keywords:  Challenges  in  authoring.  Instructional  management.  Adaptive 
instructional  systems.  Evaluation  methods.  Instructional  domain.  Delivery  of 
instruction. 

Citation:  Sottilare,  R.  A.  (2017,  April  28).  Challenges  in  Authoring,  Instructional 
Management,  and  Evaluation  Methods  for  Adaptive  Instructional  Systems.  Paper 
presented  at  American  Education  Research  Association  (AERA)  Conference 
Technology,  Instruction,  Cognition  &  Eeaming  Special  Interest  Group  (AERA 
TICE  SIG)  Symposium  on  Intelligent  Tutoring  Systems,  BIG  DATA-Eearning 
Analytics,  and  Automated  Humanlike  Tutoring:  Similarities  and  Differences  (pp. 
1-6).  San  Antonio,  TX:  Army  Research  Eab. 

Examining  Motivational  Feedback  Sensor-Free  Detected 
Frustration  Within  Game-Based  Learning 

Abstract:  Social  interactions,  decision-making,  perceptions,  and  learning  are  all 
influenced  by  affect.  Erustration,  anxiety,  and  fear  in  particular  can  draw  cognitive 
resources  away  from  successful  task  completion,  causing  the  learner  to  focus  on  the 
source  of  the  emotion  instead.  Serious  games  offer  an  ideal  environment  to 
investigate  how  feedback  influences  student  affect  and  learning  outcomes, 
particularly  when  feedback  is  delivered  via  computer  system  detection.  This 
dissertation  discusses  the  results  of  an  experiment  run  in  September  2015  to 
investigate  which  motivational  feedback  condition  yields  the  most  significant 
correlation  to  positive  learning  gains  when  a  computer  system  intelligently 
generates  and  delivers  feedback  based  on  the  detection  of  frustration  while 
participants  played  the  serious  video  game,  vMedic,  a  combat  casualty  care 
simulation  which  includes  triage  tasks.  Of  the  three  motivational  feedback 
conditions  examined  (self-efficacy,  social-identity,  and  control- value),  the  self- 
efficacy  motivational  feedback  interventions  yielded  positive,  statistically 
significant  learning  gains  when  compared  to  the  social  identity  and  control-value 
feedback  conditions,  as  well  as  the  non-motivational  feedback  control  condition, 
and  the  no  feedback  control  condition. 

Keywords:  Motivational  feedback.  Affect.  Social  interactions.  Decision  making. 
Perceptions.  Eeaming.  Successful  task  completion.  Self-efficacy  motivational 
feedback. 

Citation:  DeEalco,  J.  A.  (2017).  Examining  motivational  feedback  for  sensor-free 
detected  frustration  within  game -based  learning  (Doctoral  dissertation,  Columbia 
University). 
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Detecting  and  Addressing  Frustration  in  a  Serious  Game  for 
Military  Training 

Abstract:  Tutoring  systems  that  are  sensitive  to  affect  show  considerable  promise 
for  enhancing  student  learning  experiences.  Creating  successful  affective  responses 
requires  considerable  effort  both  to  detect  student  affect  and  to  design  appropriate 
responses  to  affect.  Recent  work  has  suggested  that  affect  detection  is  more 
effective  when  both  physical  sensors  and  interaction  logs  are  used,  and  that  context- 
sensitive  design  of  affective  feedback  is  necessary  to  enhance  engagement  and 
improve  learning.  In  this  paper,  we  provide  a  comprehensive  report  on  a  multi-part 
study  that  integrates  detection,  validation,  and  intervention  into  a  unified  approach. 
This  paper  examines  the  creation  of  both  sensor-based  and  interaction-based 
detectors  of  student  affect,  producing  successful  detectors  of  student  affect.  In 
addition,  it  reports  results  from  an  investigation  of  motivational  feedback  messages 
designed  to  address  student  frustration,  and  investigates  whether  linking  these 
interventions  to  detectors  improves  outcomes.  Our  results  are  mixed,  finding  that 
self-efficacy  enhancing  interventions  based  on  interaction-based  affect  detectors 
enhance  outcomes  in  one  of  two  experiments  investigating  affective  interventions. 
This  work  is  conducted  in  the  context  of  the  GIFT  framework  for  intelligent 
tutoring,  and  the  TC3Sim  game-based  simulation  that  provides  training  for  first 
responder  .skills. 

Keywords:  Affect  detection.  Motivational  feedback.  Game-based  learning.  GIFT. 

Citation:  DeFalco,  J.  A.,  Rowe,  J.  P.,  Paquette,  L.,  Georgoulas-Sherry,  V., 
Brawner,  K.,  Mott,  B.  W.,  ...  &  Lester,  J.  C.  (2017).  Detecting  and  addressing 
frustration  in  a  serious  game  for  military  training.  International  Journal  of  Artificial 
Intelligence  in  Education,  1-42. 

Educational  Data  Mining  Using  GIFT  Cloud 

Abstract:  Intelligent  Tutoring  Systems  (ITS),  like  human  instructors,  make 
frequent  decisions  about  what  to  present  to  the  student.  These  decisions  include 
what  courses  or  content  to  present  next,  as  well  as  what  type  of  After  Actin  Review 
(AAR)  to  present  to  the  student  after  each  course.  Ideally,  the  AAR  would  be 
Adaptive  (AAAR).  In  this  work,  we  analyze  the  decisions  of  course  content  and 
presentation.  We  construct  a  student  model  which  models  the  skill  necessary,  the 
effectiveness  of  each  course  at  training  each  skill,  and  the  relationship  between  in¬ 
scenario  measures  and  student  skill-level.  If  the  student  model  is  accurate  and 
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represented  mathematically,  then  decision-theory  can  be  used  by  the  ITS  to  select 
courses  and  course  content. 

Keywords:  Educational  data  mining.  GIFT  cloud.  Intelligent  tutor  systems.  After 
actin  review.  Adaptive  after  actin  review.  Decision  theory. 

Citation:  Carlin,  A.,  Oster,  E.,  Nucci,  C.,  Kramer,  D.,  &  Brawner,  K.  (2017,  July). 
Educational  Data  Mining  Using  GIFT  Cloud.  In  Proceedings  of  the  5th  Annual 
Generalized  Intelligent  Framework  for  Tutoring  (GIFT)  Users  Symposium 
(GIFTSymS)  (p.  253).  Robert  Sottilare. 

Establishing  Ground  Truth  on  Psychophysiological  Models  for 
Training  Machine  Learning  Algorithms:  Options  for  Ground 
Truth  Proxies 

Abstract:  One  of  the  core  aspects  of  human-human  interaction  is  the  ability  to 
recognize  and  respond  to  the  emotional  and  cognitive  states  of  the  other  person, 
leaving  human-computer  interaction  systems,  at  their  core,  to  perform  many  of  the 
same  tasks. 

Keywords:  Ground  truth.  Psychophysiological  models.  Training  machine  learning 
algorithms.  Truth  proxies. 

Citation:  Brawner,  K.,  &  Boyce,  M.  W.  (2017,  July).  Establishing  Ground  Truth 
on  Pyschophysiological  Models  for  Training  Machine  Eearning  Algorithms: 
Options  for  Ground  Truth  Proxies.  In  International  Conference  on  Augmented 
Cognition  (pp.  468-477).  Springer,  Cham. 

Expanding  Authoring  Tools  to  Support  Psychomotor  Training 
Beyond  the  Desktop 

Abstract:  Design  Recommendations  for  Intelligent  Tutoring  Systems  (ITSs) 
explores  the  impact  of  intelligent  tutoring  system  design  on  education  and  training. 
Specifically,  this  volume  examines  “Authoring  Tools  and  Expert  Modeling 
Techniques.”  The  “Design  Recommendations  book  series  examines  tools  and 
methods  to  reduce  the  time  and  skill  required  to  develop  Intelligent  Tutoring 
Systems  with  the  goal  of  improving  the  Generalized  Intelligent  Framework  for 
Tutoring  (GIFT).  GIFT  is  a  modular,  service-oriented  architecture  developed  to 
capture  simplified  authoring  techniques,  promote  reuse  and  standardization  of  ITSs 
along  with  automated  instructional  techniques  and  effectiveness  evaluation 
capabilities  for  adaptive  tutoring  tools  and  methods. 
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Keywords:  Authoring  tools.  Support  psychomotor  training.  Intelligent  tutoring 
systems.  System  design  on  education  and  training.  Design  recommendations. 
Generalized  Intelligent  Framework  for  Tutoring. 

Citation:  Sottilare,  R.  A.,  Ososky,  S.  J.,  &  Boyce,  M.  Expanding  Authoring  Tools 
to  Support  Psychomotor  Training  Beyond  the  Desktop.  Design  Recommendations 
for  Intelligent  Tutoring  Systems,  347. 

Creating  a  Team  Tutor  Using  GIFT 

Abstract:  With  the  movement  in  education  towards  collaborative  learning,  it  is 
becoming  more  important  that  learners  be  able  to  work  together  in  groups  and 
teams.  Intelligent  tutoring  systems  (ITSs)  have  been  used  successfully  to  teach 
individuals,  but  so  far  only  a  few  ITSs  have  been  used  for  the  purpose  of  training 
teams.  This  is  due  to  the  difficulty  of  creating  such  systems.  An  ITS  for  teams  must 
be  able  to  assess  complex  interactions  between  team  members  (team  skills)  as  well 
as  the  way  they  interact  with  the  system  itself  (task  skills).  Assessing  team  skills 
can  be  difficult  because  they  contain  social  components  such  as  communication 
and  coordination  that  are  not  readily  quantifiable.  This  article  addresses  these 
difficulties  by  developing  a  framework  to  guide  the  authoring  process  for  team 
tutors.  The  framework  is  demonstrated  using  a  case  study  about  a  particular  team 
tutor  that  was  developed  using  a  military  surveillance  scenario  for  teams  of  two. 
The  Generalized  Intelligent  Framework  for  Tutoring  (GIFT)  software  provided  the 
team  tutoring  infrastructure  for  this  task.  A  new  software  architecture  required  to 
support  the  team  tutor  is  described.  This  theoretical  framework  and  the  lessons 
learned  from  its  implementation  offer  conceptual  scaffolding  for  future  authors  of 
ITSs. 

Keywords:  Team  tutoring.  GIFT.  Architecture.  Evaluation. 

Citation:  Gilbert,  S.  B.,  Slavina,  A.,  Domeich,  M.  C.,  Sinatra,  A.  M.,  Bonner,  D., 
Johnston,  J., ...  &  Winer,  E.  (2017).  Creating  a  team  tutor  using  GIFT.  International 
Journal  of  Artificial  Intelligence  in  Education,  1-28. 
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Motivation  and  Research  in  Architectural  Intelligent  Tutoring 


Abstract:  It  is  well-known  that  personalised  and  adaptive  training,  such  as  from  a 
human  tutor,  is  dramatically  more  effective  than  traditional  classroom  training 
(Bloom,  1984;  VanLehn,  2011).  Due  to  a  variety  of  reasons,  however,  tutoring 
systems  are  not  yet  ubiquitous  within  the  training  market.  The  US  Army  Research 
Laboratory  is  working  to  address  this  problem  and  has  recently  published  a  series 
of  research  vector  outlines,  which  guide  research  in  the  various  areas.  The  research 
within  the  architectural  vector  naturally  exists  to  support  the  other  vectors  and  to 
investigate,  standardise,  componentise,  and  commodise  the  processes  and  functions 
of  the  various  tutoring  system  aspects.  This  paper  serves  as  an  expansion  and 
companion  to  the  similarly  named  2015  International  Defense  and  Homeland 
Security  Simulation  Workshop  paper  and  yet- to-be-published  ARL  architectural 
research  plan,  expanded  in  order  to  discuss  the  progress  made  to  date,  clarify  the 
role  of  the  architecture  in  the  research,  and  discuss  some  of  the  advantages  of  a 
unified  system  as  part  of  measuring  training  effectiveness  and  overall  system 
improvement. 

Keywords:  Adaptive  and  predictive  computer-based  training.  Intelligent  tutoring 
systems.  ITSs.  Architectural  components.  Emerging  standards. 

Citation:  Brawner,  K.,  Sinatra,  A.  M.,  &  Sottilare,  R.  (2017).  Motivation  and 
research  in  architectural  intelligent  tutoring.  International  Journal  of  Simulation 
and  Process  Modelling,  72(3-4),  300-312. 

Team  Tutoring  in  the  Generalized  Intelligent  Framework  for 
Tutoring:  Current  and  Future  Directions 

Abstract:  While  the  Generalized  Intelligent  Framework  for  Tutoring  (GIFT)  has 
been  actively  developed  over  the  past  few  years,  the  majority  of  projects  to  date 
have  focused  on  individual  learners.  A  longterm  goal  of  GIFT  is  to  support  team 
tutoring  and  provide  simultaneous  computer-based  tutoring  at  the  squad  level. 

Therefore,  efforts  in  the  Team  Modeling  vector  have  been  examining  the 
implications  of  extending  GIFT  to  team  tutoring,  and  have  laid  the  groundwork  for 
creating  team  tutors  with  GIFT. 

Keywords:  Team  tutoring.  Generalized  Intelligent  Framework  for  Tutoring.  Team 
modeling. 
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Assessment  in  Intelligent  Tutoring  Systems  in  Traditional,  Mixed 
Mode,  and  Online  Courses 

Abstract:  As  technology  has  continued  to  develop,  there  have  been  new  and 
creative  ways  to  use  it  in  the  classroom.  Classes  have  transformed  into  learning 
environments  that  rely  not  only  on  the  in-person  instructor,  but  have  many  other 
resources  easily  available  for  students.  At  the  college  level,  courses  can  vary 
between  different  modes  of  instruction:  traditional  in-person,  mixed  mode  (classes 
with  both  an  in-person  and  online  component),  and  online.  Students  routinely  bring 
laptops  and  tablets  to  class  to  both  take  notes  and  work  on  their  assignments. 
Further,  there  are  often  computer  labs  available  for  students  to  use  for  assignments 
at  universities.  In  some  cases,  these  labs  can  be  reserved  for  specific  classes  and 
provide  all  students  with  the  opportunity  to  engage  with  a  computer  during  class 
time.  Even  traditional  in-person  courses  often  have  online  materials  available  for 
students  through  learning  management  systems  (LMSs)  such  as  Blackboard  or 
Webcourses.  Students  often  engage  in  message  board  discussions  with  their 
classmates  and  download  resources  such  as  class  PowerPoints  from  the  sites. 
Grades  are  often  times  managed  digitally  and  are  available  to  students  through 
logging  into  the  system.  Many  LMSs  also  have  built  in  assignment  submission 
sections  that  in  some  cases  can  check  the  student  submitted  work  against  others  for 
plagiarism.  Many  of  these  resources  are  used  by  instructors  as  they  design  and 
supplement  their  courses.  These  resources  can  be  combined  with  in-class 
instruction  to  build  a  learning  environment  that  provides  students  a  self-regulated 
method  of  engaging  with  additional  educational  materials.  In  the  case  of  mixed¬ 
mode  or  online  course,  the  students  heavily  rely  on  these  LMSs  and  primarily 
receive  their  instruction  through  engaging  with  the  computer-based  materials.  In 
these  types  of  classes,  the  instructor’s  goals  when  developing  materials  will  be 
different  than  for  in-person  classes,  such  that  the  online  materials  will  need  to  be 
deeper  and  more  self-explanatory.  In  online  and  mixed-mode  courses,  the  instructor 
moves  from  a  lecturing  capacity  to  a  facilitator  and  subject-matter  expert  who 
engages  with  students  when  they  turn  in  assignments,  when  they  seek  help  and 
when  they  have  questions.  The  online  class  environment  is  much  more  self-directed 
and  requires  the  students  to  engage  in  self-regulated  learning  and  be  more  aware  of 
what  knowledge  that  they  have  (metacognition),  which  does  not  necessarily  come 
naturally  to  all  students  (Schraw,  1998). 
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Recommendations  for  Use  of  Adaptive  Tutoring  Systems  in  the 
Classroom  and  in  Educational  Research 

Abstract:  The  current  paper  and  presentation  provide  background  on  the  different 
uses  of  intelligent  tutoring  systems  (ITSs)  in  context  of  course  instruction,  discusses 
specific  instructor  considerations  that  are  associated  with  their  use,  and  ways  to  use 
ITSs  for  educational  research.  Instructor  considerations  include  the  time  necessary 
to  plan  prior  to  constructing  an  ITS,  the  process  of  constructing  ITS  lessons  for  use 
by  students,  the  method  in  which  students  will  interact  with  the  ITS,  approaches  to 
incorporating  ITS  use  into  classes,  and  the  information  that  instructors  would  find 
useful  to  be  output  from  the  ITS.  Specifically,  the  Generalized  Intelligent 
Framework  for  Tutoring  (GIFT),  an  open-source,  domain  independent  ITS 
framework  will  be  discussed  as  an  approach  to  creating  adaptive  tutoring  content 
for  classroom  use.  GIFT  includes  straightforward  authoring  tools  for  instructors 
and  Subject  Matter  Experts  (SMEs).  These  authoring  tools  are  powerful,  do  not 
require  a  background  in  computer  science  to  use,  and  result  in  fully  adaptive 
computer-based  lessons.  Additionally,  GIET  provides  the  flexibility  for  instructors 
to  bring  their  pre-generated  and  already  existing  instructional  material  to  the  system 
and  use  it  to  create  ITS  lessons.  The  authoring  tools  allow  the  instructor  to 
determine  the  path  of  their  lesson  and  the  components  that  their  students  will 
experience  (i.e.  surveys,  quizzes,  lesson  materials,  videos).  The  paper  includes 
details  about  the  development  of  an  instructor  dashboard  in  GIET,  ways  for  an 
instructor  to  use  GIET  for  educational  research,  and  a  discussion  of  general  output 
information  from  ITSs  that  would  be  relevant  to  instructors. 

Keywords:  Intelligent  tutoring  systems.  Classroom.  Generalized  Intelligent 
Eramework  for  Tutoring.  Educational  research.  Adaptive  tutoring. 
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Future  Directions  for  the  Generalized  Intelligent  Framework  for 
Tutoring  (GIFT) 

Abstract:  Computerized  assessment,  with  all  of  its  promised  affordances,!  has 
been  eagerly  anticipated  in  the  field  of  educational  assessment  for  some  time.  The 
technology  was  embraced  in  high-stakes  testing  as  soon  as  it  was  available,  with 
punch  cards  being  used  for  statistical  processing,  optical  scanners  processing 
multiplechoice  answer  sheets,  and  mainframe  computers  being  brought  into 
service  to  handle  the  huge  amounts  of  data  processing  requirements  of  running  an 
examination  board.  Yet,  in  2017,  we  have  not  seen  the  revolution  in  forms  of  large- 
scale  assessment  that  we  all  expected.  There  are  some  encouraging,  notable 
developments,  which  we  shall  return  to,  but  first,  we  recount  the  story  so  far  to 
understand  the  issues  that  may  have  relevance  for  the  Generalized  Intelligent 
Framework  for  Tutoring  (GIFT)  in  its  venture  into  the  field  of  computerized 
educational  assessment  in  conjunction  with  intelligent  tutoring. 

Keywords:  Generalized  Intelligent  Framework  for  Tutoring.  Computerized 
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Generalized  Intelligent  Eramework  for  Tutoring  (GIET).  Design  Recommendations 
for  Intelligent  Tutoring  System-Volume  5:  Assessment  Methods,  5,  249. 

Analyzing  Team  Training  Data:  Aspirations  for  a  GIFT  Data 
Analytics  Engine 

Abstract:  The  primary  questions  for  any  training  situation  are,  “How  well  did  the 
trainees  do?  Are  they  trained  enough?”  And  of  course,  there  are  secondary 
questions  like,  “Who  did  best/worst?”  and  “What  were  their  major  stumbling 
blocks?”  If  the  trainees  were  running  a  footrace  through  an  obstacle  course,  these 
questions  would  be  relatively  simple  to  answer.  The  measure  of  time  to  the  finish 
line  answers  “how  well”  and  “who  did  best/worst.”  The  number  of  stumbles  on  the 
obstacles  could  be  counted  as  literal  stumbling  blocks.  And  a  threshold  based  on 
previous  performers  could  be  used  to  answer  whether  they  are  trained  enough. 
Unfortunately,  real-world  training  scenarios  are  typically  much  more  complex,  and 
answering  these  simple  questions  can  be  quite  challenging.  This  paper  discusses 
some  of  these  challenges,  especially  in  a  team  training  setting 
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An  Updated  Concept  for  a  Generalized  Intelligent  Framework 
for  Tutoring  (GIFT) 

Abstract:  The  purpose  of  this  document  is  to  provide  users  of  the  Generalized 
Intelligent  Eramework  for  Tutoring  (GIET)  with  a  fundamental  understanding  of 
the  function,  goals/challenges,  and  research  activities  associated  with  the 
development  of  a  largely  domain-independent  adaptive  instructional  capability  for 
the  US  Army.  While  the  design  team  for  GIET  imagines  capabilities  beyond 
military  training,  the  focus  of  our  research  is  to  solve  the  hard  problems  and  break 
through  barriers  to  the  adoption  of  adaptive  instruction  as  a  practical  tool  for 
guiding  military  training.  Adaptive  instruction  delivers  content,  offers  feedback, 
and  intervenes  with  learners  based  on  tailored  strategies  and  tactics  with  the  goal 
of  optimizing  learning,  performance,  retention,  and  transfer  of  skills  for  both 
individual  learners  and  teams.  As  of  this  year  (2017)  the  concept  that  developed 
into  the  initial  GIET  capability  is  five  years  old.  So  much  has  been  accomplished 
over  the  last  five  years  that  the  team  saw  the  need  to  update  the  original  GIET 
concept  document  (Sottilare,  Brawner,  Goldberg,  &  Holden,  2012)  by  discussing 
research  goals  and  their  relationship  to  military  requirements.  The  remainder  of 
this  introduction  will  focus  on  military  motivation,  tools  to  evaluate  the 
effectiveness  of  candidate  adaptive  instructional  technologies  prior  to  integration 
in  GIET,  and  the  economic  motivation  for  a  GIET. 
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understanding.  Adaptive  instruction. 
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Operationalizing  the  C's  of  Teamwork  in  an  Intelligent  Tutoring 
System 

Abstract:  One  of  the  difficulties  in  creating  a  team-focused  intelligent  tutoring 
system  (ITS)  is  defining  the  measures  used  to  assess  the  team’s  performance.  While 
the  team  research  literature  offers  nine  C’s  of  teamwork  to  consider,  e.g., 
cooperation,  communication,  etc.,  it  can  also  be  difficult  to  implement  these  in  real- 
world  practice.  This  paper  reviews  the  approach  used  in  three  team  ITSs  in  which 
the  C’s  were  used,  offering  guidance  for  future  implementation  of  team  tutors. 
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A., ...  &  Kohl,  A.  (2017,  September).  Operationalizing  the  C’s  of  Teamwork  in  an 
Intelligent  Tutoring  System.  In  Proceedings  of  the  Human  Factors  and  Ergonomics 
Society  Annual  Meeting  (Vol.  61,  No.  1,  pp.  745-749).  Sage  CA:  Los  Angeles,  CA: 
SAGE  Publications. 

The  Future  of  Adaptive  Tutoring:  Wrangling  Complexity 
across  Domains,  Applications,  and  Platforms 

Abstract:  The  purpose  of  this  panel  is  to  discuss  current  directions  in  research  and 
design  of  adaptive  tutoring,  and  the  need  for  a  method  to  uniformly  describe  tutors 
within  this  growing  field.  Discussions  will  focus  on  the  increasing  complexity  of 
individual  tutors,  as  well  as  how  tutors  could  be  categorized  through  identification 
of  relevant,  constituent  parts.  A  standardized  taxonomy  would  provide  the 
foundation  for  establishing  a  quantifiable  metric  of  complexity,  which  could  then 
be  used  to  compare  vastly  distinct  tutors  to  one  another.  Applications  of  such  a 
metric  also  include  evaluating  tutor  effectiveness  with  respect  to  learning 
outcomes,  comparing  capabilities  /  usability  of  different  adaptive  tutor  authoring 
tools,  and  providing  more  accurate  estimates  of  the  time  required  to  develop  an 
hour  of  tutoring.  Individual  elements  of  tutoring  to  be  discussed  within  the  context 
of  this  framework  include  team  tutoring,  psychomotor  tutoring,  multi-platform 
architectures,  personalized  tutoring,  and  authoring  complexity. 
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Informing  the  Long-Term  Learner  Model:  Motivating  the 
Adult  Learner  (Phase  1) 

Abstract:  The  United  States  (US)  Army  has  valued  training  across  a  wide  range  of 
performance  areas  and  has  used  a  variety  of  platforms  to  instantiate  the  training 
seen  as  the  best  fit  for  the  knowledge,  skills,  and  abilities  required  for  mission 
success.  As  the  United  States  is  coming  out  of  wartime,  an  opportunity  is  available 
to  consider  the  effectiveness  and  cost  of  training  programs  and  platforms.  The 
Army  is  developing  a  framework  for  providing  personalized,  on-demand, 
computer-based  instruction  under  the  Generalized  Intelligent  Framework  for 
Tutoring  (GIFT)  program.  Of  particular  interest  for  this  effort  is  understanding  the 
intersection  of  personality  factors,  motivation,  and  reinforcers  that  are  relevant  to 
informing  the  Fong-Term  Fearner  Model.  The  approach  proposed  is  to  use  existing 
data  as  a  starting  point  for  identifying  important  correlates  of  the  aforementioned 
concepts  and  evaluating  their  application  within  a  leaming/training  environment. 
The  goal  is  to  develop  a  motivator  capability  that  can  be  incorporated  into  GIFT. 
Phase  I  focused  on  the  initial  development  of  a  Motivator  Assessment  Tool  (MAT) 
and  integration  of  the  MAT  in  GIFT  for  purposes  of  validation.  The  focus  for 
Phases  2  and  3  will  be  to  refine  the  MAT  based  upon  validation  experiments  and 
implement  that  version  as  an  adapting  strategy  in  GIFT  for  learning  in  a  determined 
task.  The  expectation  is  that  targeted  motivator  adaption  strategy  will  increase  the 
rate  and  retention  of  learning. 
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Motivation.  Thinking.  Personality.  Mental  processes.  Military  personnel.  Software 
tools. 
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Adaptive  Training  Across  Simulations  in  Support  of  a  Crawl- 
Walk-Run  Model  of  Interaction 

Abstract:  In  this  paper  we  present  a  model  of  training  interaction  based  on  tenets 
of  intelligent  tutoring  systems  and  skill  development  and  acquisition.  We  present  a 
Crawl-Walk-Run  approach  to  training  management  that  associates  with  the  utility 
of  complimentary  simulation  environments  to  build  a  set  of  skills  across  multiple 
exercises  and  scenarios.  In  addition,  we  highlight  the  role  adaptive  training 
technologies  can  play  in  this  model  of  interaction,  along  with  technologies  being 
developed  to  support  the  authoring  and  configuration  of  these  experiences.  The 
work  presented  within  is  based  on  the  Generalized  Intelligent  Framework  for 
Tutoring  (GIFT)  and  discusses  the  development  of  tools  and  methods  to  author  and 
deliver  adaptive  training  content  grounded  in  skill  development. 
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Assessing  Motivation  to  Individualize  Reinforcement  and 
Reinforcers  for  an  Intelligent  Tutor 

Abstract:  Personalized  learning  with  technology  is  in  full  demand  across  all 
context.  Learning  occurs  through  motivation  therefore,  personalizing  motivation  is 
key  to  enhancing  learning  rate  and  retention  for  the  learner.  Supplying  the 
intelligent  tutors  with  key  information  not  will  advance  the  familiarity  of 
individual’s  motivational  factors  and  interest  for  individualizing  motivation. 
Building  this  relationship  stems  from  a  streamlined  Motivational  Assessment  Tool 
(MAT),  aimed  at  assessing  several  motivation  factors.  The  Motivation  Assessment 
Tool  is  based  on  the  interconnectedness  of  motivational  factors  with  personality. 
The  creation  of  the  assessment  allows  the  intelligent  tutor  to  implement  reinforcers 
that  influence  motivational  level  based  off  individual  variances  such  as  personality. 

Keywords:  Motivational  factors.  Intelligent  tutor.  Individualized  motivation. 
Personality.  Motivational  Assessment  Tool. 


Approved  for  public  release;  distribution  is  unlimited. 


19 


Citation:  Lameier,  E.,  Reinerman-Jones,  L.,  Boyce,  M.  W.,  &  Biddle,  E.  (2017, 
July).  Assessing  Motivation  to  Individualize  Reinforcement  and  Reinforcers  for  an 
Intelligent  Tutor.  In  International  Conference  on  Augmented  Cognition  (pp.  175- 
184).  Springer,  Cham. 

Using  Assessment  to  Provide  Application  in  Human  Factors 
Engineering  to  USMA  Cadets 

Abstract:  This  paper  discusses  a  collaboration  between  the  Army  Research 
Eaboratory  (ARE)  and  the  United  States  Military  Academy  at  West  Point  in 
teaching  the  fundamentals  of  human  factors  engineering  through  assessment  and 
experimentation.  To  facilitate  this,  the  cadets  engage  in  a  year-long  capstone  project 
where  an  ARE  scientist  serves  as  a  mentor,  often  in  conjunction  with  departmental 
faculty.  This  paper  discusses  a  five-phase  teaching  process  to  assist  in  training  the 
fundamentals  of  research.  The  five  phases  are:  1.  Identification  of  research 
questions  and  background  research.  2.  The  development  of  research  protocols  and 
their  associated  training,  3.  The  selection  and  understanding  of  appropriate 
assessment  techniques,  4.  The  coordination  and  execution  of  data  collection,  and  5. 
Statistical  analysis  and  reporting.  This  process  uses  an  existing  research  experiment 
at  ARE  focusing  on  the  impact  of  different  types  of  display  surfaces  to  support 
military  tactical  decision-making  to  serve  as  a  case  study.  The  research  experiment 
used  a  hybrid  of  two  research  platforms:  the  Augmented  Reality  Sandtable  (ARES) 
and  the  Generalized  Intelligent  Eramework  for  Tutoring  (GIFT).  An  examination 
of  the  process  as  well  as  perspectives  from  the  cadets  assisting  with  the  Perspectives 
from  The  ARE/USMA  research  and  the  cadets  assisting  with  the  research.  These 
perspectives  can  help  with  the  development  of  other  similar  programs  aimed  at 
combining  research  laboratories  and  academic  institutions. 
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The  GIFT  2017  Architecture  Report 


Abstract:  The  first  version  of  the  Generalized  Intelligent  Framework  for  Tutoring 
(GIFT)  was  released  to  the  public  in  May  of  2012.  One  year  later  the  first 
symposium  of  the  GIFT  user  community  was  held  at  the  Artificial  Intelligence  and 
Education  conference  in  Memphis  Tennessee.  Since  then  the  GIFT  development 
team  has  continued  to  gather  feedback  from  the  community  regarding 
recommendations  on  how  the  GIFT  project  can  continue  to  meet  the  needs  of  the 
user  community  and  beyond.  This  current  paper  invites  and  encourages  members 
of  the  GIFT  user  community  to  continue  to  share  their  feedback,  research  findings 
and  technology  innovations  with  the  development  team  and  with  one  another  in 
order  to  strengthen  the  power,  usability  and  flexibility  of  the  GIFT  project.  As  a 
follow  up  to  the  “GIFT  2015  Report  Card  and  State  of  the  Project”  (Brawner  & 
Ososky,  2015),  and  GIFT  2016  Community  Report,  the  feature  requests  and 
responses  have  been  broken  out  among  a  number  of  papers  discussing  research 
vectors.  This  paper  discusses  the  ongoing  architectural  working  and  changes  in 
support  of  the  various  sets  of  projects. 
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Framework  for  Tutoring  (GIFT)  Users  Symposium  (GIFTSym5)  (p.  3).  Robert 
Sottilare. 

The  2017  Overview  of  the  GIFT  Authoring  Experience 

Abstract:  One  of  the  primary  goals  of  the  Generalized  Intelligent  Framework  for 
Tutoring  (GIFT)  is  to  reduce  the  time  and  skill  required  to  create  adaptive  training. 
Achieving  those  goals  within  the  GIFT  platform  is  enabled  by  a  set  of  authoring 
tools  and  associated  resources,  known  collectively  as  the  Authoring  Experience. 
The  current  paper  will  discuss  the  state  of  the  GIET  Authoring  Experience,  as  well 
as  how  feedback  and  data  from  the  community  is  informing  user-centered  design 
efforts  within  the  GIET  authoring  tools. 
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Citation:  Ososky,  S.  (2017,  July).  The  2017  Overview  of  the  GIFT  Authoring 
Experience.  In  Proceedings  of  the  5th  Annual  Generalized  Intelligent  Framework 
for  Tutoring  (GIFT)  Users  Symposium  (GIFTSymS)  (p.  39).  Robert  Sottilare. 

Focused  Authoring  for  Building  GIFT  Tutors  In  Specialized 
Domains:  A  Case  Study  Of  Psychomotor  Skills  Training 

Abstract:  As  expressed  in  the  Army  Teaming  Model  (AFM),  psychomotor  skills 
are  foundational  to  many  of  the  competencies  that  compose  the  U.S.  Army’s  vision 
for  21st  Century  Soldier  Competencies.  Training  psychomotor  skills  is  being 
addressed  in  part  through  the  use  of  sophisticated  intelligent  tutoring  systems  (ITS) 
that  tailor  and  adapt  instmction  during  simulations,  and  promising  ITS  investments 
have  been  made  in  numerous  domains  including  marksmanship  and  tactical  combat 
casualty  care.  However,  current  ITS  authoring  tools  tend  to  lack  generalization  and 
are  limited  in  scope.  The  process  to  develop  ITS  thus  remains  time-consuming  and 
costly.  For  the  Army  to  successfully  realize  the  AFM  vision,  creating  ITS  that  target 
psychomotor  skills  must  be  an  affordable,  replicable,  and  reusable  process. 

Keywords:  Generalized  Intelligent  Framework  for  Tutoring.  Case  study. 
Psychomotor  skills  training.  Simulations. 

Citation:  Bell,  B.,  Brown,  D.,  Goldberg,  B.,  &  Goldberg,  B.  S.  (2017,  July). 
Focused  authoring  for  building  GIFT  tutors  in  specialized  domains:  a  case  study  of 
psychomotor  skills  training.  In  Proceedings  of  the  5th  Annual  Generalized 
Intelligent  Framework  for  Tutoring  (GIFT)  Users  Symposium  (GIFTSym5)  (p.  51). 
Robert  Sottilare. 

Development  of  an  Integrated,  User-Friendly  Authoring  Tool  for 
Intelligent  Tutoring  Systems 

Abstract:  The  Generalized  Intelligent  Framework  for  Tutoring  (GIFT)  is  a  modular 
suite  of  capabilities  aimed  at  overcoming  the  challenges  associated  with  authoring 
and  delivering  computer-based  instruction  via  an  intelligent  tutoring  system 
(Sottilare,  Brawner,  Goldberg,  &  Holden,  2012).  One  of  the  goals  for  GIFT 
development  is  to  create  an  integrated,  user-friendly  authoring  experience  that  can 
be  used  across  training  applications.  Human  proof,  with  teammate  Design 
Interactive,  is  currently  developing  the  second-generation  GIFT  Wrap  prototype,  a 
software  application  that  allows  training  developers  to  configure  the  real-time, 
automated  delivery  of  instructional  content  triggered  by  assessing  state  changes 
within  the  training  application’s  learning  environment  (e.g.,  entity  location  within 
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a  virtual  environment)  and/or  learner  (e.g.,  progress  toward  concept  mastery, 
changes  in  cognitive  workload).  This  ongoing  research  and  development  effort  is 
focused  on  the  design  and  implementation  of  the  user  interface  that  guides  users 
through  the  set-up  of  tutoring  events.  Integration  with  ARES  (Augmented  REality 
Sandtable)  served  as  the  first  use  case.  ARES,  which  can  be  used  as  a  tactical 
training  and  mission  rehearsal  platform,  provided  the  context  for  demonstrating 
GIET  Wrap’s  utility  for  defining  real-time  assessments  directly  within  existing 
training  scenarios.  The  following  sections  briefly  describe  the  previous  version  of 
GIET  Wrap,  provide  a  detailed  discussion  of  development  efforts  to  date,  and 
present  concepts  for  extending  the  capability  to  other  training  applications  as  well 
as  live  training  environments. 

Keywords:  User  friendly  authoring  tool.  Intelligent  tutoring  systems.  Generalized 
Intelligent  Eramework  for  Tutoring.  GIET  wrap  prototype.  Augmented  REality 
Sandtable. 
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(GIET)  Users  Symposium  (GIETSymS)  (p.  77).  Robert  Sottilare. 

Learner  Models  in  the  Generalized  Intelligent  Framework  for 
Tutoring:  Current  Work  and  Future  Directions 

Abstract:  The  function  of  an  intelligent  tutoring  system  (ITS)  is  to  adapt  or  tailor 
training  to  an  individual  learner.  As  with  a  human  tutor,  this  requires  the  ITS  to 
have  some  “knowledge”  of  the  learner  (i.e.,  a  learner  model).  The  ITS  uses  and 
updates  the  learner  model  as  the  learner  progresses  through  the  material.  Eor 
example,  if  the  learner  masters  some  concept,  the  learner  model  must  be  updated  to 
reflect  this.  On  the  other  hand  if  the  learner  has  difficulty  with  a  concept,  the  ITS 
needs  to  be  able  to  understand  where  deficiencies  lie  in  order  to  prescribe  the 
appropriate  remediation.  Understanding  why  the  learner  might  have  had  difficulty 
with  a  particular  concept  is  no  simple  task  as  the  list  of  reasons  could  be  quite 
extensive.  Perhaps  the  learner  lost  focus  during  the  presentation  of  a  key  piece  of 
information,  lacks  some  key  prerequisite  knowledge,  or  has  a  low  aptitude  for  the 
domain.  The  list  could  go  on  and  on.  All  of  these  possible  explanations  require 
assessment  of  the  learner.  As  can  be  seen  from  the  above  example,  assessments 
can  include  information  about  the  learner’s  background,  experiences,  traits,  and 
aptitudes,  as  well  as  measures  of  the  learner’s  affect,  behavior,  and  performance 
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during  the  training  session.  The  more  eompletely  the  learner  model  represents  the 
learner,  the  better  the  ITS  will  be  able  to  effectively  adapt  training. 

Keywords:  Learner  models.  Generalized  Intelligent  Framework  for  Tutoring. 
Tailor  training.  Individual  learner.  Intelligent  tutor  system. 
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(GIFT)  Users  Symposium  (GIFTSymS)  (p.  89).  Robert  Sottilare. 

Multi-Level  User  Modeling  in  GIFT  to  Support  Complex  Learning 
Tasks 

Abstract:  Open-ended  computer-based  learning  environments  (OELEs)  are  user- 
centered.  They  present  users  with  complex  problems  to  solve,  and  a  set  of  tools  and 
resources  that  support  the  problem-solving  task.  While  problem  solving,  users 
typically  explore  multiple  solution  approaches,  and  assess  their  evolving  solutions 
to  make  sure  they  are  making  progress  toward  their  learning  and  problem-solving 
goals.  In  general,  OEEEs  focus  on  developing  users’  (1)  cognitive  skills,  (2) 
metacognitive  processes  and  (3)  problem  solving  strategies  that  go  beyond  the 
acquisition  of  domain- specific  cognitive  skills  (Hannafin  et  ah,  1994).  These 
environments  make  high  cognitive  demands  on  users,  and  promote  the 
development  of  strategies  and  metacognitive  processes  that  can  support  planning, 
monitoring,  and  self-evaluation  processes. 

Keywords:  Generalized  Intelligent  Eramework  for  Tutoring.  Multi-level  user 
modeling.  Complex  learning  tasks.  Open-ended  computer-based  learning 
environments. 
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Pedagogical  Management  in  Support  of  a  Generalized 
Framework  for  Intelligent  Tutoring 

Abstract:  A  current  goal  associated  with  the  development  of  the  Generalized 
Intelligent  Framework  for  Tutoring  (GIFT)  is  providing  a  set  of  tools  for  training 
practitioners  to  rapidly  build  adaptive  instructional  materials  based  on  an  interplay 
of  knowledge  acquisition  and  skill  development.  To  accommodate  this  guiding 
requirement,  an  instructional  management  research  vector  was  devised  as  a  means 
to  coordinate  resources  and  efforts  to  meet  the  needs  of  end  users.  While  the  science 
surrounding  intelligent  tutoring  system  (ITS)  development  is  multidisciplinary,  a 
guiding  assumption  is  that  targeted  GIFT  developers  in  the  military  community  will 
be  subject  matter  experts  (SMEs)  within  their  respected  fields,  but  lack  many  of  the 
technical  disciplines  that  go  into  ITS  development.  Accordingly,  it  is  also  safe  to 
assume  these  SME  developers  will  also  require  assistance  in  authoring  training 
content  that  adheres  to  learning  science  principles.  As  such,  authoring  workflows 
and  ITS  methods  must  be  developed  to  compensate  for  the  skills  a  GIET  user  lacks 
when  creating  a  lesson  or  course. 

Keywords:  Generalized  Intelligent  Eramework  for  Tutoring.  Pedagogical 
management.  Tools  for  training  practitioners.  Intelligent  tutor  systems.  Subject 
matter  experts. 
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Generalized  Intelligent  Eramework  for  Tutoring  (GIET)  Users  Symposium 
(GIFTSymS)  (p.  139).  Robert  Sottilare. 

Developing  a  Pattern  Recognition  Structure  to  Tailor  Mid- 
Lesson  Feedback 

Abstract:  The  Generalized  Intelligent  Framework  for  Training  (GIFT)  has  the 
potential  to  increase  the  micro-adaptive  individualization  of  many  training  systems 
by  overlaying  adaptive  feedback  to  learners  during  training  sessions.  For  example, 
GIFT  can  augment  a  particular  scenario  in  a  first-person  infantry  simulation  without 
needing  to  change  the  scenario  itself,  by  displaying  feedback  messages  in  the  tutor 
user  interface  (TUI)  when  particular  learner  experiences  are  observed.  Feedback 
that  GIFT  delivers  in  the  TUI  can  be  as  effective  as  feedback  embedded  directly  in 
the  system  (Goldberg  &  Cannon-Bowers,  2015 


Approved  for  public  release;  distribution  is  unlimited. 


25 


Keywords:  Pattern  recognition  structure.  Tailor  mid-lesson  feedback.  Generalized 
Intelligent  Framework  for  Tutoring.  Training  systems.  Tutor  user  interface. 

Citation:  Folsom-Kovarik,  J.  T.,  &  Boyce,  M.  W.  (2017,  May).  Developing  a 
Pattern  Recognition  Structure  to  Tailor  Mid-Lesson  Feedback.  In  Proceedings  of 
the  5th  Annual  Generalized  Intelligent  Framework  for  Tutoring  (GIFT)  Users 
Symposium  (GIFTSymS)  (p.  149).  Robert  Sottilare. 

Integrating  the  Outer  Loop:  Validated  Tutors  for  Portable 
Courses  and  Competencies 

Abstract:  The  Generalized  Intelligent  Framework  for  Tutoring  (GIFT)  is  a  broad 
and  flexible  framework  for  developing  and  delivering  training.  GIFT  currently 
provides  support  for  best  practices  and  most  effective  pedagogy  in  the  tutoring 
context,  however,  it  is  limited  in  providing  support  in  the  course  development 
process.  In  this  paper,  we  present  the  GIFT  Structural  Equation  Modeling  (GIFT- 
SEM)  module  for  doing  rapid  analysis  and  validation  of  the  concepts  and  concept 
models  used  to  guide  training  development.  We  believe  this  is  one  piece  of  a  suite 
of  validation  tools  that  will  be  used  to  allow  GIET  authors  to  identify  and  share 
effective  pedagogy,  concept  models,  sequencing,  and  learning  resources.  This 
validation  suite  will  provide  the  evaluation  necessary  to  integrate  GIFT  tutors  into 
synthetic  training  environments  (Dumanoir,  2015)  and  larger  service  oriented 
architecture  (SOA)  ecosystems.  Additionally,  validated  tutors  and  pedagogy  allow 
authors  to  learn  from  each  other’s  work,  share  validated  training  throughout  the 
GIFT  ecosystem,  leverage  near  and  far  transfer  of  learning,  and  validate  external 
resources  within  their  domain 

Keywords:  Validated  tutors.  Portable  courses.  Competencies.  Generalized 
Intelligent  Framework  for  Tutoring.  GIFT  Structural  Equation  Modeling. 
Validation  suite.  Synthetic  training  environments.  Service  orientated  architecture. 
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Approved  for  public  release;  distribution  is  unlimited. 


26 


Toward  Simulated  Students  for  Reinforcement  Learning-Driven 
Tutorial  Planning  in  GIFT 

Abstract:  A  critical  feature  of  intelligent  tutoring  systems  (ITSs)  is  their  capacity 
to  guide  and  scaffold  student  learning.  Tutorial  planners  leverage  contextual 
information  to  determine  how  pedagogical  feedback,  hints,  and  prompts  should  be 
tailored  to  learners  at  run-time  (Woolf,  2008).  Recent  years  have  witnessed  growing 
interest  in  applying  reinforcement  learning  (RL)  to  devise  tutorial  planners.  RL 
provides  a  data-driven  framework  for  creating  tutorial  planners  from  observations 
of  student  behavior  and  learning  outcomes.  RL  techniques  introduce  the  potential 
for  ITSs  that  can  automatically  refine  and  improve  their  pedagogical  methods  over 
time.  RL  methods  account  for  the  inherent  uncertainty  in  how  learners  respond  to 
different  types  of  tutorial  strategies  and  tactics,  and  they  produce  models  that  can 
be  automatically  induced  to  optimize  measures  of  student  learning.  Recent  work  on 
reinforcement  learning-based  tutorial  planning  has  outlined  a  path  for  devising 
pedagogical  models  across  a  broad  range  of  learning  environments  and  educational 
domains  (Chi,  VanLehn,  Litman,  &  Jordan,  2011;  Rowe  &  Lester,  2015;  Williams 
et  ah,  2016). 
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in  GIFT.  In  Proceedings  of  R.  Sottilare  (Ed.)  5th  Annual  GIFT  Users  Symposium. 
Orlando,  FL. 

Expanding  Domain  Modeling  in  GIFT 

Abstract:  The  purpose  of  this  paper  is  to  update  users  of  the  Generalized  Intelligent 
Framework  for  Tutoring  (GIFT;  Sottilare,  Brawner,  Goldberg,  &  Holden,  2012; 
Sottilare,  Brawner,  Sinatra,  &  Johnston,  2017,  in  press)  on  new  and  emerging 
capabilities  to  represent  a  broader  variety  of  task  domains  in  Intelligent  Tutoring 
Systems  (ITSs)  in  support  of  adaptive  instruction.  Adaptive  instruction  delivers 
content,  offers  feedback,  and  intervenes  with  learners  based  on  tailored  strategies 
and  tactics  with  the  goal  of  optimizing  learning,  performance,  retention,  and 
transfer  of  skills  for  both  individual  learners  and  teams.  GIFT  is  a  tutoring 
architecture  that  has  evolved  over  the  last  five  years  with  three  primary  goals:  1) 
reduce  the  time  and  skill  required  to  author  TTSs,  2)  automate  best  practices  of 
instruction  in  the  policy,  strategies,  and  tactics  of  tutoring,  and  3)  provide  a  testbed 
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to  assess  the  effectiveness  of  adaptive  instructional  tools  and  methods  with  respect 
to  learning,  performance,  retention,  and  transfer  of  skills.  Another  overarching  goal 
for  GIFT  has  been  to  adapt  ITSs  to  provide  instruction  in  militarily-relevant  training 
and  educational  domains 
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Modeling  Training  Efficiency  and  Return  on  Investment  for 
Adaptive  Training 

Abstract:  Adaptive  training  promises  more  effective  training  by  tailoring  content 
to  each  individual.  Where  nonadaptive  training  may  be  just  right  for  one  segment 
of  the  student  population,  there  will  be  some  students  that  find  it  too  easy  while 
others  find  it  too  difficult.  Another,  often  ignored  benefit  of  adaptive  training,  is 
improved  training  efficiency  by  minimizing  the  presentation  of  unnecessary 
material  to  learners.  One  implication  of  this  is  that  intelligent,  adaptive  training 
should  require  less  time  to  train  a  population  of  learners  to  a  given  level  of 
proficiency  than  non-adaptive  training.  The  gains  in  efficiency  should  be  a  function 
of  several  factors  including  learner  characteristics  (e.g.,  aptitude,  reading  ability, 
prior  knowledge),  learning  methods  employed  by  the  adaptive  training  system, 
course  content  (e.g.,  difficulty  and  length,  adaptability),  and  test  characteristics 
(e.g.,  difficulty,  number  of  items).  This  paper  describes  the  development  of  a 
predictive  model  for  training  efficiency  based  on  those  factors  and  how  it  could  be 
integrated  into  the  Generalized  Intelligent  Eramework  for  Tutoring  (GIET) 
architecture.  How  this  model  supports  return  on  investment  decisions  for  authors 
is  also  discussed. 

Keywords:  Modeling  training  efficacy.  Adaptive  training.  Generalized  Intelligent 
Eramework  for  Tutoring  architecture.  Investment  decisions. 
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A  Data  Analytics  Framework  to  Support  Training  Effectiveness 
Evaluation 

Abstract:  As  the  Generalized  Intelligent  Framework  for  Tutoring  (GIFT) 
framework  becomes  more  pervasive  as  a  research  tool,  the  ability  to  evaluate 
student  performance  will  be  a  critical  consideration  in  its  adoption.  Having  a 
common  method  for  evaluating  learner  performance  and  course  data  in  a  simple 
user-intuitive  way  will  aid  course  evaluators,  instructional  designers,  and  content 
managers  to  better  assess  the  effectiveness  of  a  course.  In  order  to  produce  accurate 
measurement  of  learner  performance  within  an  intelligent  tutoring  system,  a  robust 
data  analytics  framework  must  include  the  ability  to  analyze  course  performance 
data,  learner  attributes,  and  learning  strategies.  Given  the  minimal  amount  of 
existing  intelligent  tutoring  data,  a  tool  for  generating  synthetic  data  would  enhance 
the  ability  to  build  models  and  conduct  robust  experiments.  This  data  can  be  used 
to  establish  and  validate  analytic  findings,  and  develop  baselines  against  which  to 
compare  observed  performance.  This  paper  introduces  a  data  authoring  tool  and 
demonstrates  its  use. 
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Challenges  for  Assessing  and  Tutoring  Collective  Skills 

Abstract:  Computer-based  intelligent  tutoring  systems  (ITSs)  rely  on  assessment 
of  learner  behavior,  and  indeed,  ITSs  cannot  accurately  function  without  an 
“understanding”  of  the  specific  learner.  As  a  result,  it  is  of  critical  importance  to 
understand  what  can  and  cannot  be  assessed  automatically,  given  various  learning 
environments  and  learning  objectives,  as  this  will  impact  ITS  functionality.  Here, 
we  define  an  assessment  as  the  result  of  applying  a  formula  to  data  (a 
measurement),  and  then  comparing  the  score  on  the  measurement  to  an  expert 
behavior  (the  assessment).  Assessment  is  especially  challenging  in  environments 
focused  on  collective  tasks  (i.e.,  those  that  require  organized  team  or  unit 
performance  for  accomplishment;  Department  of  the  Army,  2012),  in  which  many 
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of  the  key  behaviors  of  interest  involve  coordination  and  communication  between 
human  and/or  synthetic  actors  (broadly  referred  to  as  collective  skills  here) 

Keywords:  Assessing  and  tutoring  collective  skills.  Computer  based  intelligent 
tutoring  systems.  Team  performance.  Collective  tasks. 

Citation:  Ayers,  J.,  Bink,  M.  L.,  &  Diedrich,  F.  J.  Challenges  for  Assessing  and 
Tutoring  Collective  Skills.  Design  Recommendations  for  Intelligent  Tutoring 
Systems,  287. 

Lessons  Learned  from  Large-Scale  E-Assessments:  Future 

Directions  for  the  Generalized  Intelligent  Framework  for 
Tutoring  (GIFT) 

Abstract:  Computerized  assessment,  with  all  of  its  promised  affordances,!  has 
been  eagerly  anticipated  in  the  field  of  educational  assessment  for  some  time.  The 
technology  was  embraced  in  high-stakes  testing  as  soon  as  it  was  available,  with 
punch  cards  being  used  for  statistical  processing,  optical  scanners  processing 
multiple  choice  answer  sheets,  and  mainframe  computers  being  brought  into 
service  to  handle  the  huge  amounts  of  data  processing  requirements  of  running  an 
examination  board.  Yet,  in  2017,  we  have  not  seen  the  revolution  in  forms  of  large- 
scale  assessment  that  we  all  expected.  There  are  some  encouraging,  notable 
developments,  which  we  shall  return  to,  but  first,  we  recount  the  story  so  far  to 
understand  the  issues  that  may  have  relevance  for  the  Generalized  Intelligent 
Framework  for  Tutoring  (GIFT)  in  its  venture  into  the  field  of  computerized 
educational  assessment  in  conjunction  with  intelligent  tutoring. 

Keywords:  Generalized  Intelligent  Framework  for  Tutoring.  Large  scale  E 
assessments.  Computerized  assessment.  Intelligent  tutoring. 
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Assessing  Individual  Learner  Performance  in  MOOCs 


Abstract:  Massive  open  online  courses  (MOOCs)  have  emerged  as  a  prominent 
mode  of  online  education,  but  the  quality  of  assessments  in  MOOCs  remains 
inconsistent.  There  has  been  a  consistent  gap  between  the  state  of  the  art  in 
assessment  and  the  state  of  the  practice  in  both  MOOCs  and  other  forms  of 
educational  technology.  Improving  the  quality  of  assessment  has  the  potential  to 
improve  their  usefulness  for  certification,  formative  feedback,  and  learning.  In  this 
chapter,  we  discuss  this  gap  and  efforts  to  improve  assessment  in  MOOCs.  There 
are  several  potential  directions  for  improving  assessments  in  MOOCs,  including 
improving  the  psychometric  properties  of  simple  assessment  types,  such  as 
multiple-choice  and  fill-in-the-blank  questions;  creating  richer  assessment 
experience  through  more  student  interaction  and  more  engaging  experiences  (e.g., 
games,  and  simulations),  through  leveraging  help  resources  to  see  how  students 
perform  with  some  degree  of  support;  and  using  more  powerful  technology  such  as 
automated  essay  scoring  to  better  assess  students.  We  discuss  both  existing  efforts 
and  ways  that  research  in  other  communities  can  be  incorporated  into  MOOC 
platforms 

Keywords:  Individual  learner  performance.  Massive  open  online  courses. 
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Individual  Learner  Performance  in  MOOCs.  Design  Recommendations  for 
Intelligent  Tutoring  Systems,  85. 

Principles  of  Assessment  in  the  Generalized  Intelligent 
Framework  for  Tutoring  (GIFT) 

Abstract:  The  challenges  of  assessing  human  performance  have  been  around  for 
centuries,  long  before  the  advent  of  the  Generalized  Intelligent  Framework  for 
Tutoring  (GIFT).  This  section  focuses  on  a  discussion  of  principles  of  assessment 
and  on  how  those  principles  should  be  applied  in  the  context  of  GIFT  delivered 
training.  While  concepts  like  reliability  and  validity  are  discussed,  the  goal  of  this 
section  is  not  simply  to  provide  a  review  of  basic  psychometric  terms  and  methods. 
Rather,  the  authors  have  focused  more  on  applying  lessons  learned  from  their 
respective  experiences  and  disciplines  to  make  recommendations  on  the 
development  and  design  of  GIFT.  The  chapters  examine  quite  a  range  of  topics 
from  the  somewhat  philosophical  (e.g.,  understanding  the  reasons  and  goals  of 
assessment)  to  the  very  practical  (e.g.,  developing  an  instructor  dashboard  to  make 
student  performance  measures  more  accessible  to  instructors).  Key  themes  that  run 
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through  these  chapters  include  understanding  how  evidence  and  data  can  be  used 
to  automatically  support  the  validation  of  assessments  in  GIFT,  creating  a 
standardized  method  for  implementing  assessments  in  GIFT,  and  providing 
transparency  of  assessments  to  users  of  GIFT. 

Keywords:  Generalized  Intelligent  Framework  for  Tutoring.  Principle  assessment. 
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Systems,  183. 

Assessment  of  Individual  Learner  Performance  in  Psychomotor 
Domains 

Abstract:  The  goal  of  training  is  to  support  the  learner  to  achieve  the  learning 
objectives,  i.e.,  reducing  the  task  completion  time,  reducing  errors,  and  increasing 
accuracy.  Performance  characterized  by  such  speed  and  accuracy  can  be  evaluated 
by  interpreting  the  learner  behavior  -  that  is,  as  one  of  the  classic  models,  Fitts’  law 
provides  a  way  to  assess  a  regularity  of  a  simple  psychomotor  task,  such  as  tapping 
or  pointing  (Fitts,  1954).  Based  on  the  Fitts’  model,  researchers  have  investigated 
the  coordination  of  physical  human  movement  and  information  processing.  The 
model  has  been  used  and  expanded  to  predict  the  time  to  position  and  move  around 
a  mouse  cursor  in  the  human-computer  interaction  area  (e.g.,  MacKenzie,  1992). 

Keywords:  Individual  learner  performance.  Psychomotor  domains.  Performance. 
Psychomotor  tasks. 
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Individual  Learner  Performance  in  Psychomotor  Domains.  Design 
Recommendations  for  Intelligent  Tutoring  Systems,  319. 

Coordinating  Evidence  Across  Learning  Modules  Using  Digital 
Badges 

Abstract:  No  matter  how  successful  a  learning  module  or  intervention  such  as  an 
intelligent  tutoring  system  (ITS)  is  at  producing  learning,  the  fruits  of  those  efforts 
cannot  be  employed  efficiently  without  a  suitable  means  for  representing  and 
conveying  which  learners  possess  which  skills.  Who  will  know  to  hire  or  promote 
this  more  knowledgeable  individual,  if  there  is  no  clear  sign  that  they  are  more 
accomplished?  Digital  badges  are  digital  artifacts  that  function  as  markers  of 
achievement.  Often  described  as  building  on  the  combined  traditions  found  within 
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Scouting  (e.g.,  Boy  Scouts  or  Girl  Scouts)  and  online  gaming  (e.g.,  Xbox  Live 
Achievements,  PlayStation  Network  Trophies),  badges  are  issued  to  an  individual 
when  the  individual  meets  specific  criteria  embedded  in  program-relevant  activities 
(Ostashewski  &  Reid,  2015). 

Keywords:  Learning  modules.  Digital  badges.  Intelligent  tutoring  system.  Digital 
badges.  Markers  of  achievement. 
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Cognitive  Assessment  as  Service  in  the  Generalized  Intelligent 
Framework  for  Tutoring  (GIFT) 

Abstract:  In  current  and  earlier  volumes  of  the  series,  more  detailed  analyses  of 
ITSs  have  been  presented.  In  general,  an  ITS  has  been  considered  as  an  integrated 
collection  of  more  complicated  models  and  processes  (Sottilare,  Graesser,  Hu  & 
Holden,  2013;  Sottilare,  Graesser,  Hu  &  Goldberg,  2014;  Sottilare,  Graesser,  Hu  & 
Brawner,  2015;  Sottilare,  Graesser,  Hu,  Olney,  Nye  &  Sinatra,  2016).  With  the 
limited  space  and  restricted  focus  of  the  current  chapter,  we  focus  only  on  the 
evaluative  component  of  ITSs  (step  D  of  the  outline  sequence),  especially,  we 
consider  such  component  in  the  context  of  the  General  Intelligent  Framework  for 
Tutoring  (GIFT).  For  the  purpose  of  illustration,  we  consider  two  types  of  learner’s 
responses:  categorical  responses  (such  as  multiple  choices)  and  natural  language 
responses.  For  each  of  these  two  types  of  response  types,  we  present  theory-based 
assessment  models,  implemented  as  standalone  software  service  following  the  best 
practice  of  service-oriented  architecture  (SOA).  We  argue  that  such  web  services 
can  be  used  to  serve  GIFT  and  as  an  example  of  cognitive  assessment  web  service 
for  GIFT. 

Keywords:  Cognitive  assessment.  Generalized  Intelligent  Framework  for 
Tutoring.  Intelligent  tutoring  system.  Service-orientated  architecture. 
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Toward  Systematic  Assessment  of  Human  Performance 
Interventions  in  the  US  Army:  An  Assessment  Process 
Framework 

Abstract:  The  purpose  of  this  chapter  is  to  present  some  of  the  ideas  emerging  from 
the  UMMPIREE  effort.  Specifically  to  present  a  definition  of  assessment  and  a 
framework  that  depicts  steps  regarding  how  assessment  should  be  accomplished. 
The  framework  is  applied  to  assessing  human  performance  in  technologically 
augmented  learning  environments  such  as  intelligent  tutoring  systems  (ITSs), 
specifically  the  Generalized  Intelligent  Eramework  for  Tutoring  (GIET). 

Keywords:  Systematic  assessment.  Human  performance  intervention.  Intelligent 
tutoring  system.  US  army. 
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Performance  Interventions  in  the  US  Army:  An  Assessment  Process  Eramework. 
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Assessment  of  Forgetting 

Abstract:  Most  efforts  to  educate  individuals  in  a  domain  conclude  with  efforts  to 
assess  the  effectiveness  of  the  educational  activities.  During  education  research,  the 
assessment  activity  tends  to  focus  on  the  performance  of  the  students  and  what  that 
reflects  on  the  quality  of  the  educational  intervention.  After  a  quality  educational 
intervention  has  been  deployed,  the  focus  of  assessment  turns  to  the  individual 
students.  In  both  cases,  there  are  multiple  considerations  in  collecting,  analyzing, 
and  interpreting  the  results  of  assessments. 
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Motivating  Individual  Differences  in  an  Intelligent  Tutoring 
System 

Abstract:  A  key  ingredient  of  achievement,  engagement,  and  learning  is 
motivation.  Motivated  trainees  believe,  value,  focus  on  learning,  manage  a  task  or 
time  more  efficiently,  and  persist.  The  flip-side  results  in  disengagement, 
procrastination,  anxiety,  loss  of  control,  negative  thoughts,  failure,  or  a  complete 
shutdown.  The  benefits  to  properly  assessing  and  individually  motivating  learners 
are  saving  lives,  reducing  cost,  saving  time,  and  increasing  retention.  The  challenge 
is  determining  a  goodness  of  fit  for  each  individual  that  is  measured  by  an  increase 
in  effort,  attention,  goal  attainment,  learning  outcomes,  and  retention  within  an 
intelligent  tutor.  Traditionally,  trainers  develop  relationships  that  paint  a  clear 
picture  of  the  individual’s  motivation,  but  class  size  restricts  individualization. 
Intelligent  tutors  have  the  potential  to  assess  in  real  time,  plan,  and  implement 
individualized  motivational  strategies  to  a  task  lacking  luster. 

Keywords:  Individual  differences.  Intelligent  tutoring  system.  Intelligent  tutors. 
Motivation. 
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Assessment  in  Intelligent  Tutoring  Systems  in  Traditional,  Mixed 
Mode,  and  Online  Courses 

Abstract:  As  technology  has  continued  to  develop,  there  have  been  new  and 
creative  ways  to  use  it  in  the  classroom.  Classes  have  transformed  into  learning 
environments  that  rely  not  only  on  the  in-person  instructor,  but  have  many  other 
resources  easily  available  for  students.  At  the  college  level,  courses  can  vary 
between  different  modes  of  instruction:  traditional  in-person,  mixed  mode  (classes 
with  both  an  in-person  and  online  component),  and  online.  Students  routinely  bring 
laptops  and  tablets  to  class  to  both  take  notes  and  work  on  their  assignments. 
Eurther,  there  are  often  computer  labs  available  for  students  to  use  for  assignments 
at  universities.  In  some  cases,  these  labs  can  be  reserved  for  specific  classes  and 
provide  all  students  with  the  opportunity  to  engage  with  a  computer  during  class 
time. 

Keywords:  Assessment  of  courses.  Intelligent  tutoring  systems.  Traditional  course. 
Mixed  course.  Online  course.  Performance  differences. 
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Introduction  to  Assessments  Methods  &  GIFT 

Abstract:  This  book  is  the  fifth  in  a  planned  series  of  books  that  examine  key  topics 
(e.g.,  learner  modeling,  instructional  strategies,  authoring,  domain  modeling, 
assessment,  impact  on  learning,  team  tutoring,  machine  learning,  and  potential 
standards)  in  intelligent  tutoring  system  (ITS)  design  through  the  lens  of  the 
Generalized  Intelligent  Framework  for  Tutoring  (GIFT)  (Sottilare,  Brawner, 
Goldberg  &  Holden,  2012;  Sottilare,  Brawner,  Sinatra,  &  Johnston,  2017).  GIFT  is 
a  modular,  service-oriented  architecture  created  to  reduce  the  cost  and  skill  required 
to  author  ITSs,  manage  instruction  within  ITSs,  and  evaluate  the  effect  of  ITS 
technologies  on  learning,  performance,  retention,  transfer  of  skills,  and  other 
instructional  outcomes.  Along  with  this  volume,  the  first  four  books  in  this  series. 
Learner  Modeling  (ISBN  978-0-9893923-0-3),  Instructional  Management  (ISBN 
978-0-9893923-2-7),  Authoring  Tools  (ISBN  978-0-9893923-6-5)  and  Domain 
Modeling  (978-0-9893923-9-6)  are  freely  available  at  www.GIFTtutoring.org  and 
on  Google  Play.  This  introduction  begins  with  a  description  of  tutoring  functions, 
provides  a  glimpse  of  assessment  best  practices,  and  examines  the  motivation  for 
standards  in  the  design,  authoring,  instruction,  and  evaluation  of  ITS  tools  and 
methods.  We  introduce  GIFT  design  principles  and  discuss  how  readers  might  use 
this  book  as  a  design  tool.  We  begin  by  examining  the  major  components  of  TTSs. 

Keywords:  Generalized  Intelligent  Framework  for  Tutoring.  Assessment  methods. 
Introduction  to  GIFT. 
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Understanding  Competency  Assessment  Methodologies  Applied 
to  Adaptive  Instruction  in  Intelligent  Tutoring  Systems 

Abstract:  The  first  goal  of  this  chapter  is  to  provide  the  reader  with  a  basic 
understanding  of  competence,  competency  assessment  methods,  and  their  relationship 
to  processes  leading  to  knowledge  and  skill  acquisition  during  adaptive  instruction. 
The  second  goal  is  to  introduce  the  chapters  in  this  section  of  the  book  and  define  each 
chapter’s  importance  to  competency  assessment  and  the  design  of  Intelligent  Tutoring 
Systems  (TTSs).  Let’s  begin  by  defining  competence  and  competency.  In  terms  of 
learning,  competence  is  an  “ability  or  skill”  or  “the  ability  to  do  something  successfully 
or  efficiently”  (Merriam-Webster,  2017).  The  goal  of  adaptive  instruction  is  to  guide 
the  learner  in  developing  competence  in  one  or  more  domains. 

Keywords:  Assessment  methodologies.  Adaptive  instruction.  Intelligent  tutoring 
systems.  Introduction  to  ITSs. 
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Resting  Evidence  in  Assessment  and  Intelligent  Tutors 

Abstract:  Assessment  design  methodologies  such  as  evidence-centered  design 
(BCD)  provide  an  approach  for  representing  the  “argument”  underlying  an 
assessment.  Argument-based  structures  articulate  the  chain  of  reasoning  connecting 
task-level  data  to  the  evidence  required  to  support  assessment  claims  about  the 
student.  Intelligent  tutoring  systems  (ITSs)  reflect  similar  underlying  rationale, 
with  decisions  about  how  to  update  the  student  model  and  what  to  present  to  the 
student  next,  dependent  on  student  performance  (evidence).  Although  the  concept 
of  “evidence”  is  widely  used  within  the  assessment  community,  it  is  less  well 
articulated  in  the  ITS  literature.  In  this  chapter,  we  argue  for  the  necessity  of 
formalizing  the  concept  of  evidence  and  for  implementing  tools  and  services  that 
facilitate  the  use  of  evidence  in  ITSs.  Explicitly  representing  evidence  may  provide 
a  stronger  theoretical  basis  for  an  ITS  and  facilitate  the  reuse  of  assessment 
components  across  ITSs.  Furthermore,  additions  to  the  Generalized  Intelligent 
Framework  for  Tutoring  (GIFT)  may  support  defining  links  from  observable 
evidence  to  claims  (number  of  links,  types  of  evidence,  types  of  observables,  and 
properties)  as  well  as  mechanisms  for  quantifying  evidence.  Several  examples 
illustrate  how  this  approach  facilitates  reuse  and  handling  of  evidence  in  the  areas 
of  conversation-based  assessment  and  GIFT-medical  prototypes 
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Using  Mobile  Technology  to  Generate  Learning  Content  for  an 
Intelligent  Tutoring  System 

Abstract:  With  the  release  of  the  “Army  Eeaming  Concept  for  2015,”  the  Army 
signaled  a  need  to  change  the  way  it  trained.  One  goal  was  to  reduce  the  amount  of 
lecture-based  classes,  and  move  towards  “...more  engaging  technology-delivered 
instruction  that  will  be  used  as  part  of  a  blended  learning  approach,  distributed  to 
the  workforce  for  job-related  sustainment  learning,  and  as  performance  support 
applications.”  As  a  result,  a  repository  of  learning  modules  will  be  needed  to 
support  career  progression,  assignment-oriented  learning,  operational  lessons,  and 
performance  support.  In  addition,  intelligent  tutors,  generated  with  the  Generalized 
Intelligent  Eramework  for  Tutoring  (GIET),  will  tailor  the  learning  experience  to 
the  individual  learner. 

Our  research  examined  effective  ways  for  guiding  users,  e.g.  subject  matter  experts, 
instructional  systems  designers,  etc.  in  the  development  of  reusable  learning  objects 
(REOs)  to  support  adaptive  learning.  REOs  are  small  “chunks”  of  instructional 
content  which  are  associated  with  learning  objectives.  The  research  included 
studying  the  common  structure  of  REOs  and  defining  approaches  for  creating  REOs 
via  easy-to-use,  guided  user  interactions.  In  addition,  a  prototype  authoring  system 
was  developed  to  provide  a  robust  set  of  capabilities  for  quick  and  easy 
development  of  effective  REOs,  leveraging  the  capability  of  mobile  devices  to 
create  rich,  multi-media  assets.  The  resulting  Android-based  mobile  app:  1)  guides 
users  through  the  development  of  instructionally  sound  REOs  and  lessons  that  are 
created  from  sequencing  multiple  REOs;  2)  provides  support  for  including 
multimedia  assets  such  as  video,  audio,  still  imagery,  and  text  to  convey 
knowledge,  3)  supports  the  development  of  embedded  assessments  within  REOs, 
and  4)  saves/packages  the  instructional  content  in  a  sharable  format  (as  a  sharable 
content  object).  This  paper  discusses  the  user  needs  analysis,  requirements 
definition,  authoring  tool  prototype,  exemplar  training  content  development,  and 
results  from  preliminary  usability  and  instructional  system  design  evaluations. 
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GIFT  Cloud:  Improving  Usability  of  Adaptive  Tutor  Authoring 
Tools  Within  a  Web-Based  Application 

Abstract:  GIFT  Cloud  is  the  reeently  released  web-based  application  version  of 
GIFT,  an  open-source  computer-based  tutoring  architecture  that  supports 
authoring,  deployment,  and  evaluation  of  intelligent  tutoring  system  technologies. 
This  paper  presents  the  GIFT  Cloud  Authoring  Tools,  through  the  lens  of  usability. 
Each  major  element  within  the  authoring  tools  is  described,  along  with  usability 
design  considerations  that  were  made  in  order  to  reduce  occurrence  of  error,  to 
organize  information,  and  to  support  end-user  goals.  The  initial  release  of  GIFT 
Cloud  supports  an  iterative  design  approach,  informed  by  user  data  and  feedback, 
with  an  overall  goal  of  making  tutor  authoring  practical  for  subject  matter  experts 
without  computer  programming  or  instructional  design  knowledge.  As  such, 
lessons  learned  from  this  release,  as  well  as  plans  for  future  research  and  usability 
improvements,  are  discussed. 
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The  Effects  of  Self-Reference  and  Context  Personalization  on 
Task  Performance  During  Adaptive  Instruction 

Abstract:  An  advantage  of  computer-based  instruction  is  that  student  entered 
information  can  be  saved  and  used  throughout  learning.  Self-reference  (tying 
information  to  the  self)  has  been  shown  to  have  a  positive  impact  on  memory  and 
learning.  This  study  evaluates  the  impact  of  including  self-reference  and  familiar 
popular  culture  names  during  the  assessment  phase  of  adaptive  instruction. 
Participants  engaged  with  a  computer-based  tutorial  about  solving  logic  grid 
puzzles  and  were  assessed  by  completing  additional  puzzles.  The  assessment 
puzzles  included  the  participant’s  and  friends’  names  (self-reference),  popular 
culture  names,  or  generic  names.  Participants  in  the  popular  culture  condition  spent 
significantly  less  time  solving  the  standard  puzzle  than  those  in  the  generic 
condition,  with  no  difference  in  percentage  correct.  The  inclusion  of  popular  culture 
names  may  have  facilitated  more  efficient  task  performance  while  maintaining 
quality  of  performance.  It  is  envisioned  that  this  strategy  can  be  implemented  in 
computer-based  adaptive  instruction  to  improve  task  efficiency. 

Keywords:  Computer-based  instruction.  Self-reference.  Adaptive  instruction.  Task 
efficiency. 
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adaptive  instruction.  In  Proceedings  of  the  Human  Factors  and  Ergonomics  Society 
Annual  Meeting  (Vol.  60,  No.  I,  pp.  398-402).  Sage  CA:  Los  Angeles,  CA:  SAGE 
Publications. 

Agent-Based  Practices  for  an  Intelligent  Tutoring  System 
Architecture 

Abstract:  The  Generalized  Intelligent  Framework  for  Tutoring  (GIFT)  project  is 
partially  an  effort  to  standardize  the  systems  and  processes  of  intelligent  tutoring 
systems.  In  addition  to  these  efforts,  there  is  emerging  research  in  agent-driven 
systems.  Agent-based  systems  obey  software  and  messaging  communication 
protocols  and  accomplish  objectives  to  the  original  system,  but  have  different 
architectural  structure.  This  paper  describes  the  upcoming  research  changes  for 
GIFT,  from  a  module-driven  system  to  an  agent-driven  system,  the  reasons  for 
wanting  to  do  so,  the  advantages  of  the  change,  some  initial  technical  approaches 
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which  encapsulate  current  functionality,  and  the  types  of  research  that  this  change 
will  enable  in  the  future. 
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Conference  on  Augmented  Cognition  (pp.  3-12).  Springer  International  Publishing. 

Examining  the  Influence  of  Heartbeat  on  Expert  Marksman 
Performance 

Abstract:  Marksmanship  training  in  the  US  Army  follows  a  process  that  all 
marksmen  must  learn,  understand,  and  experience  to  improve  their  shot.  This 
standard  can  be  broken  down  into  key  factors  over  which  a  marksman  has  control 
and  needs  to  develop  to  improve  his  or  her  marksmanship.  For  that  reason,  in  this 
technical  note  we  examine  the  role  of  the  heartbeat  as  observed  from  expert 
marksmen.  The  influence  of  the  heartbeat  is  categorized  into  3  different  areas:  1)  if 
expert  marksmen  time  the  shot  on  a  heartbeat,  2)  if  they  fire  between  heartbeats,  or 

3)  if  they  simply  ignore  their  heartbeat  during  marksmanship.  The  data  support  that 
expert  marksmen  ignore  their  heartbeat  when  firing.  Because  of  this  evaluation  of 
expert  behavior,  we  determine  the  heartbeat  to  be  a  nonfactor  in  marksmanship 
training. 

Keywords:  Training.  Heart  rate.  Marksmanship.  Instructions.  Behavior. 
Performance  human. 
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Modelling  a  Learner's  Affective  State  in  Real  Time  to  Improve 
Intelligent  Tutoring  Effectiveness 

Abstract:  This  paper  introduces,  describes,  and  evaluates  real-time  models  of 
affective  states  of  individual  learners  interacting  with  Intelligent  Tutoring  Systems. 
Computer-based  instructors,  like  human  instructors,  should  use  affective 
information  for  adapting  instruction.  This  requires  an  accurate  representation  of 
individual  learner  state  during  tutoring;  however,  models  described  in  the  literature 
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are  generalized  and  constructed  offline.  Such  total  population  models  have  faced 
validation  difficulty  with  individuals,  while  individualized  models  have  had 
difficulties  with  offline  creation  and  online  use.  The  simultaneous  creation  and 
utilization  of  an  individualized  model  from  sensor-based  physiological 
measurements  presents  an  attractive  alternative.  We  present  and  evaluate 
approaches  for  building  affective  models  during  the  tutoring  session  which  address 
the  difficulties  present  in  real-time  data  streams.  Additionally,  this  work  examines 
the  impact  of  occasional  direct  user  query  on  model  quality.  The  results  indicate 
that  individualized  real-time  model  construction  is  comparable  to  offline 
equivalents,  yet  can  be  successfully  applied  in  tutoring  settings. 

Keywords:  Affective  learner  state.  Real-time  systems.  Computer- managed 
instruction.  Artificial  intelligence.  Intelligent  tutoring  systems. 
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Considerations  for  Immersive  Learning  in  Intelligent  Tutoring 
Systems 

Abstract:  Research  has  examined  the  benefits  and  retractors  of  immersing  the 
learner  in  an  environment.  Immersive  computer-based  training  environments  are 
costly  to  construct  and  may  not  always  lead  to  significant  learning  or  transfer 
benefits  over  other  methods.  The  current  paper  presents  a  brief  review  of  presence 
and  immersion  research  in  computer-based  learning  and  adaptive  tutoring.  The 
Generalized  Intelligent  Framework  for  Tutoring  (GIFT)  is  an  open  source  domain- 
independent  framework  for  creating  intelligent  tutoring  systems  (ITS).  GIFT  offers 
flexibility,  and  can  be  interfaced  with  training  applications  ranging  from  highly 
immersive  computer-based  learning  environments  (e.g.,  TC3Sim,  VBS2)  to  less 
immersive  mediums  such  as  PowerPoint.  The  capabilities  of  GIFT  that  can  be  used 
to  create  immersive  adaptive  tutoring  are  discussed.  Additionally,  the  use  of  GIFT 
to  run  and  generate  experimental  studies  to  examine  the  impact  of  immersion  is 
highlighted.  Finally,  recommendations  are  given  on  how  to  provide  more 
opportunities  to  integrate  immersive  environments  into  GIFT. 

Keywords:  Immersion.  Intelligent  tutoring  systems.  Generalized  Intelligent 
Framework  for  Tutoring.  Presence. 
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A  Review  of  Self-Reference  and  Context  Personalization  in 
Different  Computer-Based  Educational  Domains 

Abstract:  A  goal  of  instruction  is  for  a  student  to  not  only  learn  the  material  and 
be  able  to  recall  it  immediately  afterwards,  but  also  to  be  able  to  retain  it  for  future 
use.  Additionally,  it  is  preferable  for  the  material  to  be  taught  in  such  a  way  that  the 
student  can  use  what  was  learned  and  then  apply  or  transfer  it  to  new  situations. 
Human  memory  and  learning  have  been  extensively  studied,  resulting  in 
mnemonics  and  general  strategies  that  assist  in  memory  across  different  domains 
of  instruction.  One  such  strategy  is  linking  in  formation  to  knowledge  that  the 
student  already  has  and  that  is  important  to  them.  This  strategy  can  include  linking 
information  to  the  self,  to  people  that  the  individual  knows,  or  simply  customizing 
the  materials  to  include  mention  of  topics  that  are  of  interest  to  the  individual  or 
group  of  students.  There  appears  to  be  significant  benefits  to  employing  all  of  these 
approaches;  however,  they  are  often  lumped  together  in  the  literature,  and  the 
different  domains  in  which  they  may  or  may  not  be  beneficial  have  not  been 
acknowledged. 

Many  different  domains,  topics,  and  areas  of  instruction  include  materials, 
examples,  and  practice  problems  with  wording  that  can  be  adjusted  to  make  it 
specific  to  different  students  or  interest  areas.  In  many  classes,  whether  they  are 
lecture-based,  web-based,  large  or  small,  it  is  often  difficult  to  retain  student  interest 
and  motivation  throughout.  However,  there  are  different  approaches  that  can  be 
used  to  draw  student  attention  and  personalize  the  materials  that  they  receive  in 
order  to  keep  them  more  interested  in  the  topic.  Referencing  the  self  and  context 
personalization  (changing  the  wording  of  student  materials  to  be  contextually 
consistent  with  student  interests)  has  been  shown  to  have  positive  impacts  on 
student  out  comes.  Self-reference  and  context/interest  personalization  techniques 
offer  benefits  to  memory  and  performance  without  requiring  large  adjustments  to 
learning  materials.  This  chapter  reviews  research  and  approaches  to  self-reference, 
context  personalization,  interest  personalization,  and  the  impacts  that  they  have  on 
learning  and  retention  in  different  domains.  It  has  been  assumed  that  these  strategies 
will  work  between  learning  domains;  however,  research  has  focused  primarily  on 
mathematics.  The  domains  of  instruction  are  examined  and  recommendations  are 
made  for  using  this  strategy  in  intelligent  tutoring  systems  (ITSs). 
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Exploring  the  Diversity  of  Domain  Modeling  for  Training  and 
Educational  Applications 

Abstract:  Domain  models  represent  the  knowledge  within  a  task  domain  in  order 
to  facilitate  instruction.  This  knowledge  may  be  visual  (e.g.,  text,  graphic)  or  verbal 
and  represents  domain  content  to  be  presented  to  the  learner  during  instruction, 
feedback  provided  by  the  tutor,  and  other  communications  initiated  by  the  tutor 
(e.g.,  hints,  prompts,  questions,  assertions).  The  goal  of  this  chapter  is  to  explore 
assorted  domain  models  as  they  are  applied  within  intelligent  tutoring  systems 
(ITSs)  during  adaptive  instruction.  Domains  may  be  defined  per  the  various 
taxonomies  in  the  literature:  cognitive  (Bloom,  1956),  affective  (Krathwohl,  Bloom 
&  Masia,  1964),  psychomotor  (Simpson,  1972),  or  social  (Seller,  2001),  but  most 
instructional  domains  are  a  mixture  of  two  or  more  of  these  taxonomies. 

Within  the  Generalized  Intelligent  Framework  for  Tutoring  (GIFT),  the  domain 
model  specifies  concepts  (also  known  as  learning  objectives)  and  their  associated 
methods  of  measurement  and  sources  for  data  to  determine  whether  learners  have 
met  the  standards  for  a  concept.  The  GIFT  domain  model  also  has  data  structures 
that  allow  the  author  to  specify  the  tutor  actions  (e.g.,  ask  the  learner  a  question  or 
change  the  challenge  level  of  the  environment)  in  response  to  conditions  identified 
within  the  environment  and  the  learner.  The  authored  tutor  actions  are  linked  to 
strategies  (plans)  recommended  by  the  pedagogical  model. 

Keywords:  Domain  modeling.  GIFT.  Fearning  objectives.  Pedagogical  model. 
Intelligent  tutoring  systems. 
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The  Hidden  Challenges  of  Team  Tutor  Development 


Abstract:  This  paper  describes  the  unexpected  challenges  of  team  tutor 
development  such  as  the  task  and  logistics.  Previously,  a  research  team  from  Iowa 
State  University  (ISU)  working  with  the  U.S.  Army  Research  Laboratory  (ARL) 
developed  the  reconnaissance  (Recon)  task  for  simple  team  tutoring  with  the 
Generalized  Intelligent  Framework  for  Tutoring  (GIFT)  (Bonner  et  ah,  2015; 
Gilbert  et  ah,  2015).  Considerations  were  included  for  the  testing  environment  such 
as  audio-based  team  interactions,  initialization  of  the  scenario  simultaneously,  and 
the  inclusion  of  eyetracking  and  screen  capture  technology.  Throughout  the  process 
of  tutor  development,  several  computational  challenges  have  been  encountered 
such  as  the  implementation  of  team  rules,  determination  of  the  appropriate  amount 
of  feedback,  and  the  use  of  participants’  behavior  history  as  input  to  the  tutor.  Our 
descriptions  of  these  challenges  should  forewarn  future  developers  of  team  tutors. 
We  also  suggest  enhancements  to  GIFT  to  aid  this  process. 

Keywords:  Team  tutor.  Behavior.  GIFT.  Testing  environments.  Evetracking. 
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Generalized  Intelligent  Framework  for  Tutoring  (GIFT) 
Cloud/Virtual  Open  Campus  Quick-Start  Guide 

Abstract:  This  document  serves  as  the  quick-start  guide  for  GIET  Cloud,  the  web- 
based  application  version  of  the  Generalized  Intelligent  Eramework  for  Tutoring 
(GIET).  GIET  is  a  modular,  open-source  framework  for  building,  deploying,  and 
managing  adaptive  training  content.  GIFT  Cloud  allows  learners,  authors,  and 
researchers  to  access  GIFT  without  downloading  and  installing  software  on  a 
desktop  or  laptop  computer.  The  purpose  of  this  document  is  to  provide  an 
overview  of  the  primary  interfaces  and  basic  functions  of  GIFT  Cloud.  Topics 
include  establishing  a  GIFT  Account,  taking  courses  on  GIFT  Cloud,  orienting  the 
user  to  GIFTS  tools,  and  locating  resources  and  providing  feedback  on  GIFT  Cloud. 
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Sensor-Free  or  Sensor-Full:  A  Comparison  of  Data  Modalities  in 
Multi-Channel  Affect  Detection 

Abstract:  Computational  models  that  automatically  detect  learners'  affective  states 
are  powerful  tools  for  investigating  the  interplay  of  affect  and  learning.  Over  the 
past  decade,  affect  detectors— which  recognize  learners'  affective  states  at  run-time 
using  behavior  logs  and  sensor  data-have  advanced  substantially  across  a  range  of 
K-12  and  postsecondary  education  settings.  Machine  learning-based  affect 
detectors  can  be  developed  to  utilize  several  types  of  data,  including  software  logs, 
video/audio  recordings,  tutorial  dialogues,  and  physical  sensors.  However,  there 
has  been  limited  research  on  how  different  data  modalities  combine  and 
complement  one  another,  particularly  across  different  contexts,  domains,  and 
populations.  In  this  paper,  we  describe  work  using  the  Generalized  Intelligent 
Framework  for  Tutoring  (GIFT)  to  build  multi-channel  affect  detection  models  for 
a  serious  game  on  tactical  combat  casualty  care.  We  compare  the  creation  and 
predictive  performance  of  models  developed  for  two  different  data  modalities:  1) 
software  logs  of  learner  interactions  with  the  serious  game,  and  2)  posture  data  from 
a  Microsoft  Kinect  sensor.  We  find  that  interaction-based  detectors  outperform 
posture-based  detectors  for  our  population,  but  show  high  variability  in  predictive 
performance  across  different  affect.  Notably,  our  posture-based  detectors  largely 
utilize  predictor  features  drawn  from  the  research  literature,  but  do  not  replicate 
prior  findings  that  these  features  lead  to  accurate  detectors  of  learner  affect. 
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Effect  of  Topography  on  Learning  Military  Tactics-Integration  of 
Generalized  Intelligent  Framework  for  Tutoring  (GIFT)  and 
Augmented  REality  Sandtable  (ARES) 

Abstract:  This  report  discusses  the  findings  of  a  research  project  that  integrated 
the  Generalized  Intelligent  Framework  for  Tutoring  and  the  Augmented  REality 
Sandtable  (ARES)  for  the  assessment  of  military  tactics.  An  experiment  involving 
19  Reserve  Officers  Training  Corps  cadets  from  a  large  southeastern  university 
assessed  performance,  physiological,  and  experiential  data  in  a  between-subjects 
design.  The  conditions  consisted  of  a  2-D  map  displayed  on  either  a  flat  or 
contoured  surface,  both  of  which  leveraged  the  projection  technology  of  ARES. 
Results  of  the  study  did  not  indicate  significant  differences  between  time  on  task, 
accuracy,  or  electrodermal  activity,  but  a  larger  sample  size  is  needed  to  verify 
findings.  Preference  between  conditions  was  more  prevalent  in  support  of  ARES; 
however,  individuals  that  preferred  the  flat  condition  discussed  issues  with  the 
ARES  condition  being  more  difficult  to  precisely  measure,  consistent  with  the 
findings  in  the  literature.  This  study  is  being  refined  for  a  follow-on  experiment  to 
be  conducted  at  the  United  States  Military  Academy  at  West  Point. 
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Electrodermal  Activity  Analysis  for  Training  of  Military  Tactics 

Abstract:  This  study  examined  the  differences  in  the  electrodermal  activity  (EDA) 
of  19  ROTC  cadets  viewing  a  2D  or  3D  perspective  display.  Skin  conductance 
responses  (SCRs)  were  analyzed  using  a  multistep  visual  analytic  process  and 
coded  according  to  participant  and  condition.  Results  show  non-significant 
differences  SCRs  between  conditions,  however  since  it  is  only  an  initial  study,  more 
research  needs  to  be  done  before  definitive  conclusions  can  be  made.  Euture 
investigations  involving  mixed  method  research  design  are  discussed  with  a 
specific  emphasis  on  user  engagement  and  workload.  The  results  of  this  initial  study 
will  be  incorporated  into  future  research  with  the  United  States  Military  Academy 
at  West  Point. 
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Electrodermal  Activity  Analysis  for  Training  of  Military  Tactics.  In  Proceedings  of 
the  Human  Eactors  and  Ergonomics  Society  Annual  Meeting  (Vol.  60,  No.  1,  pp. 
1339-1343).  Sage  CA:  Eos  Angeles,  CA:  SAGE  Publications. 

Elements  of  Adaptive  Instruction  for  Training  and  Education 

Abstract:  This  paper  discusses  critical  elements  of  adaptive  instruction  in  support 
of  training  and  education.  Modeling  and  assessing  learners  and  teams,  optimizing 
adaptive  instructional  methods,  applying  domain  modeling  outside  of  traditional 
training  and  educational  domains,  automating  authoring  processes,  and  assessing 
the  learning  effect  of  instruction  are  among  the  challenges  reviewed. 

Keywords:  Adaptive  instruction.  Intelligent  tutoring  systems.  Eeamer  modeling. 
Domain  modeling.  Authoring  tools.  Teaming  effect. 

Citation:  Sottilare,  R.  A.,  &  Boyce,  M.  W.  (2016).  Elements  of  Adaptive 
Instmction  for  Training  and  Education.  In  Schmorrow,  D.  D.,  &  Eidopiastis,  C.  M. 
(Eds.).  (2016).  Eoundations  of  Augmented  Cognition:  Neuroergonomics  and 
Operational  Neuroscience:  10th  International  Conference,  AC  2016,  Held  as  Part 
of  HCI  International  2016,  Toronto,  ON,  Canada,  July  17-22,  2016,  Proceedings 
(Vol.  9744).  Springer. 

Interpretative  Phenomenological  Analysis  for  Military  Tactics 
Instruction 

Abstract:  This  experiment  determined  how  19  Army  Reserve  Officer  Training 
Corps  (ROTC)  cadets  rationalize  military  tactics  decisions  using  two  different 
displays  through  qualitative  data  analysis.  Interpretative  Phenomenological 
Analysis  (IPA)  was  used  to  understand  platoon  and  squad  level  decision-making  to 
help  influence  the  development  of  adaptive  training  systems.  IPA  is  a  process 
through  which  the  participant  experience  is  analyzed  through  a  dual  interpretation 
methodology,  where  participants  provide  meaning  of  their  world,  followed  by  an 
empirical  assessment  to  categorize  and  define  the  participants’  perspectives  of  their 
experiences.  The  tactical  questions  covered  topics  that  would  typically  be  covered 
in  a  military  science  junior  level  class.  Results  include  the  importance  of  terrain 
elevation  differences  between  friendly  and  enemy  forces,  as  well  as  the  importance 
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of  cover  and  concealment  and  distance.  The  findings  from  this  experiment  are 
currently  being  developed  into  a  large  scale  assessment  in  collaboration  with  the 
United  States  Military  Academy  at  West  Point. 

Keywords:  Interpretative  phenomenological  analysis.  Military  population. 
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Phenomenological  Analysis  for  Military  Tactics  Instruction.  In  Advances  in  Human 
Factors,  Business  Management,  Training  and  Education  (pp.  623-634).  Springer 
International  Publishing. 

Automated  Detection  of  Cognitive  and  Metacognitive  Strategies 
for  Learner  Modeling  in  GIFT 

Abstract:  Promoting  students’  learning  of  cognitive  and  metacognitive  strategies 
that  may  generalize  across  domains  is  increasingly  seen  as  an  important  component 
of  intelligent  tutoring  systems  (ITSs),  especially  those  that  support  open-ended 
complex  problem  solving  and  decision  making.  Such  open-ended  learning 
environments  (OELEs)  allow  learners  to  make  choices  in  their  approach  to 
developing,  monitoring,  and  managing  their  evolving  solution  paths  (Segedy, 
Kinnebrew  &  Biswas,  2015).  To  be  successful,  learners  have  to  become  adept  at 
employing  cognitive,  metacognition  and  self-regulation  processes  and  strategies  in 
developing  their  solutions  (Butler  &  Winne,  1995;  Kinnebrew,  Segedy  &  Biswas, 
2016;  Zimmerman  &  Schunk,  2001).  Such  processes  and  strategies  typically 
encompass  information  acquisition,  situation  awareness,  plan  development  and 
refinement  taking  into  account  resource  limitations  and  trade-offs,  solution 
monitoring,  evaluation,  and,  finally,  reflection. 

In  this  paper,  we  present  our  work  on  modeling  students  learning  about 
counterinsurgency  (COIN)  operations  with  UrbanSim  (McAlinden,  et  ah,  2009),  a 
turn-based  game  environment,  where  users  take  on  the  role  of  a  battalion 
commander  to  deal  with  fictional  counterinsurgency  scenarios.  We  track  student 
problem  solving  and  analyze  student  performance  using  the  extensions  of  the  GIET 
tracking  and  learner  modeling  capabilities  that  we  are  implementing  to  develop 
metacognitive  tutoring  in  GIET.  The  analysis  of  tum-by-turn  student  performance 
is  a  first  step  toward  analyzing  students’  metacognitive  and  problem  solving 
processes.  The  data  we  analyze  in  this  paper  are  data  logged  by  UrbanSim  collected 
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in  a  study  conducted  with  Reserve  Officers’  Training  Corps  (ROTC)  officers-in- 
training  at  a  major  U.S.  University.  We  analyze  students’  operations  in  the  context 
of  the  state  of  the  simulation.  We  present  our  analysis  methods,  and  discuss  how 
the  results  will  help  us  define  learner  models  that  capture  students  cognitive  and 
metacognitive  processes. 

Keywords:  Cognitive  strategies.  Metacognitive  strategies.  Self-regulation.  Learner 
modeling.  GIFT.  UrbanSim.  Intelligent  tutoring  system.  Open-ended  learning 
environments. 
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Automated  Detection  of  Cognitive  and  Metacognitive  Strategies  for  Learner 
Modeling  in  GIFT.  In  Generalized  Intelligent  Framework  for  Tutoring  (GIFT) 
Users  Symposium  (GIFTSymd)  (p.  15). 

Elements  of  a  Learning  Effect  Model  to  Support  an  Adaptive 
Instructional  Framework 

Abstract:  This  paper  describes  the  evolution  of  a  learning  effect  model  (LEM; 
Sottilare,  2012;  Sottilare,  Ragusa,  Hoffman  &  Goldberg,  2013)  to  guide  adaptive 
instruction  within  the  Generalized  Intelligent  Framework  for  Tutoring  (GIFT; 
Sottilare,  Brawner,  Goldberg  &  Holden,  2012),  an  open-source  architecture  for 
authoring,  delivering,  guiding,  and  evaluating  tailored,  computer-based  instruction 
for  individual  learners  and  teams  of  learners.  Effect  models  may  be  used  to 
demonstrate  the  relationship  between  the  learner,  the  computer-based  tutor,  and  the 
instructional  environment  and  how  they  influence  each  other  during  adaptive 
instruction. 

The  GIET  EEM  is  focused  on  how  these  relationships  facilitate  desirable  learning 
outcomes  (e.g.,  knowledge  and  skill  acquisition,  performance,  retention,  and 
transfer  of  skills  from  training  to  the  work  environment).  The  EEM  discussed  in 
this  paper  began  as  a  set  of  strategies  (plans)  and  tactics  (actions)  used  by  a  GIET- 
generated  tutor  to  drive  tailored  training  experiences  in  real-time.  The  EEM  has 
since  evolved  to  encompass  real-time  and  long-term  models  of  both  individual 
learners  and  teams  of  learners  plus  required  knowledge,  learning  objectives, 
tailored  learning  events,  and  measures  of  learning  and  performance.  This  paper 
provides  a  detailed  breakout  of  elements  and  processes  that  compose  the  EEM  and 
a  description  of  their  function  in  the  process  of  adaptive  instruction. 


Approved  for  public  release;  distribution  is  unlimited. 


50 


Keywords:  Learner  effect  model.  Adaptive  instruction.  GIFT.  Open-sourced 
architecture.  Computer-based  instruction.  Individual  learners.  Team  learning. 
Models.  Instructional  environment. 

Citation:  Sottilare,  R.  (2016,  July).  Elements  of  a  Learning  Effect  Model  to 
Support  an  Adaptive  Instructional  Eramework.  In  Generalized  Intelligent 
Eramework  for  Tutoring  (GIET)  Users  Symposium  (GIETSymd)  (p.  7). 

Motivational  Feedback  Messages  as  GIFT  Interventions  to 
Frustration 

Abstract:  Determining  how  to  effectively  respond  to  learner  affect  is  important  not 
only  in  face-to-face  learning  environments  (Pekrun,  Goetz,  Titz  &  Perry,  2002),  but 
also  within  the  field  of  intelligent  tutoring  systems  (ITSs)  (Goldberg  et  ah,  2012). 
This  requires  not  only  tools  to  accurately  identify  affect,  but  also  developing  a  suite 
of  accompanying  interventions  that  can  respond  to  learner  affect  (D’Mello,  Lehman 
&  Graesser,  2011). 

In  an  effort  to  help  learners  regulate  their  affective  states,  some  computer  tutoring 
systems  researchers  have  used  interventional  feedback  messages  to  motivate  the 
learner  through  a  frustrated  state  (Robison,  McQuiggan  &  Lester,  2009).  However, 
these  researchers  have  noted  that  where  frustration  has  been  detected  and  feedback 
delivered,  learners  do  not  always  respond  positively  to  these  interventions,  but 
rather  may  react  negatively  to  feedback  provided  by  the  system  (Robison, 
McQuiggan  &  Lester,  2009).  This  has  given  rise  to  the  need  to  take  a  closer 
examination  of  the  design  of  motivational  feedback  messages  delivered  to  learners 
in  a  frustrated  state  to  determine  the  most  effective  approach  for  addressing  learner 
frustration  via  interventional  feedback  messages. 

Within  this  context,  then,  the  gap  addressed  by  this  current  work  is  an  effort  in 
determining  what  kind  of  motivational  feedback  messages  delivered  within  an  ITS 
effectively  addresses  the  affective  state  of  frustration  within  a  simulation-based 
training  game  and  promotes  learning  gains.  Three  theories  of  motivation  were 
targeted  to  design  feedback  messages:  1)  theory  of  control-value  (Pekrun,  Elliot  & 
Maier,  2006);  2)  theory  of  social  identity  (Tajfel  &  Turner,  1979);  and  3)  theory 
of  self-efficacy  (Bandura,  1977).  These  theories  are  distinct  from  each  other  in  the 
way  they  target  either  a  person’s  sense  of  what  they  value  (control-value  theory), 
who  they  are  (social  identity  theory),  and  what  a  person  believes  they  can  achieve 
(self-efficacy  theory). 
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The  Use  of  Social  Media  for  Creating  and  Improving  Learning 
Content 

Abstract:  The  US  Army  trains  and  edueates  over  a  half  million  individuals  per  year 
in  a  eourse-based,  throughput-oriented  system.  Mueh  of  the  Army’s  web-based 
instruetion  is  in  the  form  of  statie  PowerPoint  presentations,  with  little  tailoring  to 
individual  soldier  needs.  With  the  ever-ehanging  landseape  of  full  speetrum 
operations,  today’s  soldiers  are  faeing  ill-struetured  problems  and  have  little  time 
for  the  ideal  levels  of  refleetion  and  repetition  needed  to  promote  eritieal  thinking, 
adaptability,  and  mastery  of  eomplex  skills.  Additionally,  the  eurrent  time  frame 
for  updating  eourses  (3  to  5  years)  does  not  support  the  modem  Army’s  fast-paeed 
learning  needs. 

In  pursuit  of  more  powerful  training  tools,  the  US  Army  Researeh  Laboratory 
(ARL)  has  sponsored  researeh  resulting  in  the  Generalized  Intelligent  Framework 
for  Tutoring  (GIFT;  Sottilare,  Brawner,  Goldberg  &  Holden,  2012;  Sottilare, 
Holden,  Goldberg  &  Brawner,  2013),  an  open  souree  arehiteeture  to  lower  the  skills 
and  time  needed  to  author,  deliver,  and  evaluate  adaptive  instmetion.  To  enhanee 
the  eontent  authoring  and  management  eapabilities  of  GIFT  and  other  instruetional 
frameworks,  ARL  has  sponsored  researeh  into  a  Soeial  Media  Framework  (SMF) 
that  enables  organizations  to  erowd-souree  and  erowd-vet  new  learning  eontent  and 
improvements  to  existing  eourses.  The  researeh  questions  we  seek  to  answer  in  our 
eurrent  researeh  inelude  the  extent  to  whieh  the  SMF  and  GIFT  ean:  (a)  promote 
eritieal  thinking,  eollaboration,  adaptability,  effeetive  eommunieation,  and  problem 
solving;  (b)  help  elose  the  gap  between  formal  training  and  operational  applieation 
of  the  training  to  missions  in  the  field;  (e)  reduee  the  time  required  to  loeate  and 
use  learning  resourees;  (d)  reduee  the  time  required  to  ineorporate  feedbaek  from 
the  field  into  formal  instruetion;  and  (e)  reduee  instmetor  workload,  while 
maximizing  the  effieaey  of  the  instmetor’ s  time. 
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The  GIFT  2016  Community  Report 

Abstract:  The  first  version  of  the  Generalized  Intelligent  Framework  for  Tutoring 
(GIFT)  was  released  to  the  public  in  May  2012  (Sottilare,  Brawner,  Goldberg  & 
Holden).  One  year  later,  the  first  symposium  of  the  GIFT  user  community  was  held 
at  the  Artificial  Intelligence  and  Education  conference  in  Memphis,  Tennessee. 
Since  then,  the  GIFT  development  team  has  continued  to  gather  feedback  from  the 
community  regarding  recommendations  on  how  the  GIFT  project  can  continue  to 
meet  the  needs  of  the  user  community  and  beyond.  This  paper  continues  the 
conversation  with  the  GIFT  user  community  in  two  important  ways.  First,  it  invites 
and  encourages  members  of  the  GIFT  user  community  to  continue  to  share  their 
feedback,  research  findings,  and  technology  innovations  with  the  development 
team  and  with  one  another  in  order  to  strengthen  the  power,  usability,  and  flexibility 
of  the  GIFT  project.  This  year,  the  title  of  the  current  paper  has  been  updated  to 
reflect  the  emphasis  on  the  GIFT  community.  Second,  as  a  follow  up  to  the  “GIFT 

2015  Report  Card  and  State  of  the  Project”  (Brawner  &  Ososky,  2015),  this  paper 
briefly  describes  how  the  GIFT  development  team  is  addressing  features  requested 
in  previous  GIFT  Symposium  meetings  and  serves  as  documentation  for  the  next 
major  project  direction. 

The  research  and  technology  innovation  efforts  presented  in  the  current  document 
include  those  that  are  informed  by  the  GIFT  user  community,  and  only  represent  a 
fraction  of  the  overall  research,  development,  and  implementation  work  associated 
with  GIFT.  We  invite  the  reader  to  review  the  other  chapters  in  this  volume, 
publications  on  GIFTTutoring.org,  and  other  references  described  below,  to  get  a 
sense  of  the  total  body  of  work  on  the  GIFT  project.  Major  themes  in  this  current, 

2016  GIFT  community  discussion  include  user  experience  improvements  for 
authors  and  researchers,  functionality  supporting  experimentation  with  GIFT,  and 
the  alpha  release  of  GIFT  Cloud  as  an  enabling  technology. 
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Using  GIFT  Wrap  to  Author  Domain  Assessment  Models  with 
Native  Training  Applications 

Abstract:  A  generalized  tool  to  build  tutors  helps  to  facilitate  authoring  across 
domains,  however  it  decreases  the  likelihood  that  the  authoring  experience  will  be 
well  guided  with  context  sensitive  help  when  necessary.  For  example,  a  course 
designer  creating  a  course  on  counter  insurgency  and  another  creating  a  course  on 
crowd  control  would  benefit  from  course  creation  interfaces  tailored  for  each 
domain  to  help  guide  them  through  the  authoring  process.  However,  since  it  is 
currently  inconceivable  to  develop  an  authoring  environment  for  all  domains  and 
user  roles,  an  alternative  solution  is  needed  that  can  provide  the  domain  familiarity 
while  still  offering  a  generalized  approach.  This  paper  provides  an  insight  into 
ongoing  development  on  the  next  generation  GIFT  authoring  tool  called  GIFT 
Wrap.  GIFT  Wrap  aims  to  merge  the  abstract  nature  of  authoring  domain 
assessment  models  with  the  native  training  applications  that  subject-matter  experts 
are  familiar  with.  This  connection  will  diminish  the  learning  curve  associated  with 
authoring  for  an  ITS. 

Keywords:  Generalized  tool.  Tutors.  Authoring.  Modeling  domains.  GIFT  Wrap. 
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Extending  GIFT  with  a  Reinforcement  Learning-Based  Framework 
for  Generalized  Tutorial  Planning 

Abstract:  In  this  paper,  we  describe  our  recent  work  on  a  modular  reinforcement 
learning  framework  for  tutorial  planning  in  the  Generalized  Intelligent  Framework 
for  Tutoring  (GIFT;  Sottilare,  Brawner,  Goldberg  &  Holden  2012).  We  focus  on 
inducing  tutorial  planning  models  directly  from  learner  data  to  support  a  broad 
range  of  tutorial  interventions,  which  share  a  generalized  encoding  of  instructional 
strategies  and  tactics  across  multiple  learning  environments.  This  work  is  part  of  a 
collaborative  project  between  North  Carolina  State  University  (NCSU),  Intelligent 
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Automation,  Inc.  (lAI),  and  the  U.S.  Army  Research  Laboratory  (ARL)  to 
investigate  generalizable  data-driven  tutorial  planning  that  operates  across  multiple 
training  environments.  We  are  investigating  modular  reinforcement  learning-based 
tutorial  planning  in  the  domain  of  counterinsurgency  and  stability  operations 
(COIN)  training,  with  a  focus  on  adaptive  hypermedia  and  simulation-based 
learning  environments.  We  describe  the  initial  design  and  development  of  a 
generalized  tutorial  planner,  whose  design  is  inspired  by  Chi’s  ICAP  framework 
(2009),  which  differentiates  between  passive,  active,  constructive,  and  interactive 
forms  of  learning.  In  addition,  we  describe  a  pilot  study  that  was  conducted  with 
university  ROTC  cadets,  which  was  designed  to  test  the  impact  of  ICAP-inspired 
tutorial  interventions  on  learning  outcomes  during  COIN  training.  We  conclude 
with  a  discussion  of  design  recommendations  for  GIFT  to  facilitate  reinforcement 
learning-based  tutorial  planning  induced  from  simulated-  and  human-student  data. 
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Integrating  an  Interoperable  Competency  Model  with  GIFT  using 
the  Experience  API  (xAPI) 

Abstract:  A  robust  student  model  is  key  to  successful  intelligent  tutoring. 
Traditionally,  student  models  are  specific  to  the  intelligent  tutoring  system  (ITS)  in 
which  they  reside,  and  are  not  reusable  across  different  educational  technologies. 
As  the  U.S.  Army  works  toward  the  goal  of  maintaining  persistent  representations 
of  an  individual  learner  and  integrating  Soldier  training  across  multiple  systems, 
student  models  should  be  flexible  to  support  their  use  across  multiple  training 
platforms.  Work  on  automating  student  model  generation  from  systems  could 
provide  a  means  to  extend  the  capability  of  the  Generalized  Intelligent  Framework 
for  Tutoring  (GIFT)  and  allow  an  ecosystem  of  ITSs  and  tools  to  create  common 
value. 

In  this  paper,  we  describe  our  work  to  create  such  a  learner  model  which  we  call  an 
Interoperable  Competency  Model  (ICM)  that  can  be  used  by  GIFT.  This  model 
incorporates  xAPI  performance  data  from  training  delivered  both  within  and 
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outside  of  GIFT,  to  model  learner  competencies.  An  ICM  database  exists  outside 
of  GIFT  to  facilitate  the  data  being  accessed,  updated,  and  used  by  multiple  training 
systems.  In  this  way,  GIFT  can  effectively  cooperate  with  other  training  systems 
within  a  learning  ecosystem  to  develop  learner  competencies.  This  might  take  the 
form  of  providing  programmed  or  remedial  training,  providing  diagnosis  of  learner 
problems,  recommending  training  to  learners  and  instructors,  or  providing 
predictions  of  skill  acquisition  and/or  retention. 

Keywords:  Student  model.  Competency  model.  GIFT.  Experience  API.  Intelligent 
tutoring  systems.  Reusability.  Interoperable  competency  model.  Performance  data. 
Training. 
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M.  J.  (2016,  July).  Integrating  an  Interoperable  Competency  Model  with  GIFT 
using  the  Experience  API  (xAPI).  In  Generalized  Intelligent  Eramework  for 
Tutoring  (GIET)  Users  Symposium  (GIETSymd)  (p.  99). 

An  Adaptive  AAR  Capability  for  GIFT 

Abstract:  The  US  Army  Research  Eaboratory  (ARE)  seeks  to  improve  the 
capability  for  customized  feedback  within  GIET,  in  support  of  the  AEM  vision.  In 
this  paper,  we  report  on  progress  in  developing  an  Adaptive  After- Action  Review 
(AAR)  Module  that  is  grounded  in  the  well-validated  theory  of  deliberate  practice, 
which  suggests  that  expertise  grows  best  when  the  learner  focuses  study  and 
practice  on  specific,  deficient  knowledge  and  skills,  and  receives  feedback 
concerning  the  efficacy  of  their  performance  (Ericsson,  Krampe,  &  Tesch-Romer, 
1993).  The  AAR  model  will  improve  the  learner’s  competence  on  specific 
competencies.  It  is  based  on  a  domain-general  mathematical  approach,  and  contains 
adjustable  parameters  that  will  be  adjusted  by  looking  across  instances  in  which 
feedback  has  been  applied  and  assessing  the  impact  of  that  feedback  on  learning. 
This  model  will  address  both  components  of  the  deliberate  practice  formula,  as  it 
will  focus  study  on  specific  competencies  and  optimize  feedback  that  drives 
learning  and  improves  performance.  The  Module  will  plug  into  the 
GIETarchitecture,  making  it  simpler  for  ITS  developers  to  dynamically  optimize 
feedback  for  users  of  GIET-based  ITSs. 

The  Adaptive  AAR  Module  is  designed  to  support  the  delivery  of  AAR  during  self- 
guided  learning  and  in  classroom  instruction.  Initially,  we  are  developing  our 
technology  to  interact  with  Newton’s  Playground  (Zhao,  W.,  M.  Ventura,  et  ah. 
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2015)  in  the  physics  tutoring  domain.  The  capability  within  will  be  generic,  and  in 
future  work  will  be  expanded  to  other  domains. 

Keywords:  Customized  feedback.  GIFT.  Adaptive  after-action  review.  Feedback. 
Domain.  Model.  Self-guided  learning.  Newton’s  Playground. 
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(2016,  July).  An  Adaptive  AAR  Capability  for  GIFT.  In  Generalized  Intelligent 
Framework  for  Tutoring  (GIFT)  Users  Symposium  (GIFTSymd)  (p.  1 1 1). 

From  Concept  to  Publication-A  Successful  Application  of  Using 
GIFT  from  the  Ground  Up 

Abstract:  The  idea  for  this  paper  comes  from  the  desire  to  help  people  who  are 
unfamiliar  with,  or  novices  using  the  Generalized  Intelligent  Framework  for 
Tutoring  (GIFT,  Sottilare,  Brawner,  Goldberg,  &  Holden,  2012)  understand  how 
GIFT  can  effectively  gather  and  collect  data  without  large  amounts  of  programming 
or  intelligent  tutoring  experience.  By  providing  guidance  for  building  an 
assessment  for  an  experiment,  a  potential  author  can  give  GIFT  a  try  without 
intimidation.  This  paper  serves  as  a  companion  to  previous  papers  on 
experimentation  and  GIFT  (Goldberg  &  Cannon-Bowers,  2015;  Sinatra,  2014)  as 
well  as  papers  discussing  the  authoring  process  using  GIFT  (Brawner  &  Ososky, 
2015).  Sinatra  (2014)  discusses  GIFT  from  the  perspective  of  a  research 
psychologist.  She  explains  the  use  of  the  Course  Authoring  Tool  (CAT),  the 
creation  of  surveys  using  the  Survey  Authoring  System  (SAS),  and  the  exporting 
of  data  using  the  Event  Reporting  Tool  (ERT)  to  collect  data.  Brawner  &  Ososky 
(2015)  discuss  the  ongoing  development  with  GIET  as  a  result  of  user  requirements 
or  feature  requests.  The  paper  discusses  GIET  as  a  content  manager,  where  it  assists 
in  the  interaction  with  the  Augmented  REality  Sandtable  (ARES,  Amburn,  Vey, 
Boyce  &  Mize,  2015). 

Keywords:  GIET.  Authoring.  Experiment  assessment.  Survey  Authoring  System. 
Event  Reporting  Tool.  Data  collection.  User  requirements.  Content  management. 
ARES. 

Citation:  Boyce,  M.  W.  Erom  Concept  to  Publication-A  Successful  Application  of 
Using  GIET  from  the  Ground  Up.  In  Generalized  Intelligent  Eramework  for 
Tutoring  (GIET)  Users  Symposium  (GIFTSym4)  (p.  125). 
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The  Updated  Research  Psychologist's  Guide  to  GIFT 


Abstract:  DON’T  PANIC.  Those  two  words  were  made  famous  by  the  book  The 
Hitchhiker’s  Guide  to  the  Galaxy  (Adams,  1979),  as  their  being  on  the  cover  was 
one  of  reasons  that  the  fictitious  guide  had  sold  more  copies  than  the  Encyclopedia 
Galactica  (it  was  also  slightly  cheaper).  Those  classic  words  were  also  included  at 
the  beginning  of  my  original  Research  Psychologist’s  Guide  to  GIFT  paper 
(Sinatra,  2014)  that  was  presented  at  GIFTSym2.  Having  “Don’t  Panic”  at  the  top 
of  the  page  is  meant  to  serve  as  a  reminder  that  even  though  things  can  seem 
overwhelming  at  times  when  using  software  such  as  the  Generalized  Intelligent 
Framework  for  Tutoring  (GIFT;  Sottilare,  Brawner,  Goldberg  &  Holden,  2012), 
there  are  documents  like  this  one  that  include  explanations  on  how  to  easily  and 
efficiently  use  GIFT.  In  a  world  filled  with  movie  sequels  and  prequels  this  guide 
serves  as  a  sequel  of  sorts  to  the  original  Research  Psychologist’s  Guide  to  GIFT. 
However,  it  does  not  necessarily  replace  it.  The  information  provided  in  it  is  still 
helpful,  however,  many  changes  have  occurred  in  the  past  2  years  and  the  current 
work  provides  information  about  how  to  use  updated  features  in  GIFT,  and 
considerations  that  should  be  made  as  a  result  of  the  changes.  This  guide  itself  will 
be  tightly  coupled  with  the  releases  surrounding  it,  GIFT  2015-2X,  GIFT 
Cloud/GIFT  Virtual  Open  Campus,  and  the  soon  to  be  released  GIFT  2016-1. 
Perhaps  the  current  guide  is  a  middle  piece  of  a  trilogy  (or  a  series  of  trilogies),  and 
hopefully  future  editions  will  cover  the  new  advancements  that  are  made  with  GIFT 
over  time. 

Keywords:  Research  psychology.  Ease.  Effectiveness.  User  capabilities.  GIFT. 
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Designing  the  User  Experience  of  the  GIFT  Cloud  Authoring  Tools 

Abstract:  This  paper  presents  a  new  user  experience  for  the  authoring  tools 
associated  with  the  web-based  version  of  the  Generalized  Intelligent  Framework 
for  Tutoring  (GIFT;  Sottilare,  Brawner,  Goldberg  &  Holden,  2012),  known  as 
GIFT  Cloud.  The  new  user  experience  endeavors  to  provide  user  interfaces  (UI) 
and  interaction  paradigms  that  are  more  congruent  with  human  mental  models  of 
adaptive  tutor  authoring,  compared  to  the  system  representation  model  upon  which 
prior  versions  of  the  authoring  tools  were  more  closely  associated. 
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Based  on  a  flowchart-interaction  metaphor,  the  new  authoring  tools  will  allow 
authors  to  take  an  object  oriented  approach  to  sequencing  course  content.  The  new 
design  is  intended  to  be  learnable  and  efficient  for  authors  without  computer 
programming  or  instructional  design  experience.  This  current  paper  will  describe 
the  evolution  of  the  GIFT  authoring  tools,  articulate  the  vision  for  the  new 
authoring  experience,  and  discuss  the  principles  upon  which  the  new  design  is 
based.  This  work  is  intended  to  benefit  novice  and  intermediate  users  of  the  GIFT 
authoring  tools,  and  will  be  of  value  to  researchers  and  practitioners  interested  in 
developing  or  managing  adaptive  training  content  using  GIFT. 

Keywords:  User  experience.  Authoring  tools.  GIFT.  User  interfaces.  Adaptive 
tutor  authoring.  Adaptive  training  content. 
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Scaling  Across  Domains  and  the  Implications  for  GIFT 

Abstract:  Currently,  the  Generalized  Intelligent  Framework  for  Tutoring  (GIFT) 
adapts  training  in  a  somewhat  closed  system.  That  is,  current  GIFT  applications  are 
only  concerned  with  modeling  the  learner  and  do  main  within  the  confines  of  a 
single  block  of  training  delivered  exclusively  through  the  GIFT  framework.  While 
this  approach  has  been  advantageous  from  the  point  of  view  of  providing 
experimental  control  while  exploring  GIFT’S  utility  and  effectiveness,  it  only 
partially  addresses  what  we  know  to  be  the  “real  world”  requirements  of  an  adaptive 
training  system.  Adaptive  training  systems  like  GIFT  will  be  expected  to  function 
within  a  rich  ecosystem  of  training  events  and  include  classroom  lectures,  group 
discussions,  and  hands-on  training,  as  well  as  simulation,  gaming,  other  computer- 
based  training,  and  training  out  comes.  A  single  course  might  easily  include  all  of 
these  forms  of  training,  while  GIFT  may  be  responsible  for  only  a  subset.  For  GIFT 
to  be  most  effective  in  such  an  environment,  it  will  have  to  be  able  to  model  the 
learner  and  domain  at  a  higher  level  and  not  just  for  the  specific  blocks  of  training 
that  it  is  responsible  for  delivering. 

Language  training  and  marksmanship  training  are  real-world  examples  of  the  type 
of  training  described  above.  Neither  language  proficiency  nor  marksmanship 
expertise  develops  through  training  in  single  applications  or  over  short  periods  of 
time.  Rather,  both  require  hundreds  of  hours  of  deliberate  practice  after  mastering 
the  basics.  Training  for  both  includes  lecture,  computer-based  training,  and  live 
practice.  While  technology  has  been  shown  to  help  language  learning  (Zhao,  2003) 
and  marksmanship  training  (Chung  et  ah,  2011),  studies  are  typically  isolated  to  a 
single  application  over  a  short  period  of  time  (although  there  are  some  exceptions. 
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notably  Adair-Hauck,  Willingham-McLain  &  Youngs,  2000;  Green  &  Youngs, 
2001;  Spain  et  al.,  2013).  This  chapter  explores  the  challenges  of  scaling  the  domain 
model  to  encompass  an  entire  course,  specifically  through  the  examination  of  these 
two  training  domains. 

Keywords:  Scaling.  Domain  model.  Adaptive  application.  Training.  Course 
examination.  Learner  model.  High  level  domain. 
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Domain  Modeling  in  a  Psychomotor  World:  A  Marksmanship  Use 
Case 

Abstract:  Recent  technological  advancements  have  extended  computer-based 
training  and  education  practices  be  yond  the  traditional  desktop  environment  and 
into  domains  requiring  physical  interaction  to  conduct  tasks.  In  this  chapter,  we 
examine  a  psychomotor  use  case  as  it  relates  to  domain  modeling  within  the 
Generalized  Intelligent  Framework  for  Tutoring  (GIFT),  and  present  considerations 
to  address  when  developing  any  system  of  this  nature.  GIFT  is  a  domain- 
independent  architecture  developed  for  the  purpose  of  building  adaptive  training 
functions  and  intelligent  tutoring  systems  (ITSs)  across  an  array  of  tasks  and  a 
conceivably  unlimited  set  of  knowledge,  skills,  and  abilities  (KSAs).  A  current 
objective  is  to  apply  GIFT  tools  and  methods  in  an  Army-valued  skill  domain  that 
incorporates  psychomotor  components  of  task  execution  and  KSA  development 
(Sottilare,  Sinatra,  Boyce  &  Graesser,  2015).  The  intention  is  to  broaden  ITS 
modeling  techniques  beyond  the  traditional  cognitive  dimensions  of  learning.  In 
this  instance,  individuals  are  reliant  upon  knowledge  and  skills  associated  with 
psychomotor  functions  rather  than  cognitive  application  used  to  solve  problems. 
From  an  adaptive  training  and  ITS  perspective,  this  new  psychomotor  paradigm 
requires  a  shift  in  how  the  domain  space  is  modeled  to  associate  this  new  type  of 
input  for  informing  assessment.  Rather  than  modeling  and  monitoring  steps  toward 
solving  a  problem  and  identifying  misconceptions  and  impasses  along  the  way,  the 
psychomotor  use  case  focuses  on  behavior  and  its  inherent  influence  on 
performance;  specifically,  what  nuances  of  a  task  are  dictated  by  variations  in 
patterns  of  behavior  and  what  strategies  can  be  enacted  to  assist  an  individual  in 
acquiring  the  ability  to  replicate  a  desired  behavior  across  multiple  trials.  The  end- 
state  goal  is  to  enhance  training  systems  to  support  physical  skill  development 
through  deliberate  practice  techniques  (Ericsson,  2006,  2014)  pedagogically 
managed  by  ITS  assessment  and  feedback  methods. 
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Citation:  Goldberg,  B.,  &  Ambum,  C.  Domain  Modeling  in  a  Psychomotor  World: 
A  Marksmanship  Use  Case.  Design  Recommendations  for  Intelligent  Tutoring 
Systems,  195. 

Defining  the  Ill-Defined:  From  Abstract  Principles  to  Applied 
Pedagogy 

Abstract:  Attempts  to  define  ill-defined  domains  in  intelligent  tutoring  system 
(ITS)  research  has  been  approached  a  number  of  times  (Foumier-Viger,  Nkambou 
&  Nguifo,  2010;  Lynch,  Ashley,  Pinkwart  &  Aleven,  2009;  Mitrovic  & 
Weerasinghe,  2009;  Jacovina,  Snow,  Dai  &  McNamara,  2015;  Woods,  Stensrud, 
Wray,  Haley  &  Jones,  2015).  Related  research  has  tried  to  determine  levels  of  ill- 
definedness  for  a  domain  (Le,  Loll  &  Pinkwart,  2013).  Despite  such  attempts,  the 
field  has  not  yet  converged  on  common  guidelines  to  distinguish  between  well- 
versus  ill-defined  domains.  We  argue  that  such  guidelines  struggle  to  converge 
because  a  domain  is  too  large  to  meaningfully  categorize:  every  domain  contains  a 
mixture  of  well-  and  ill-defined  tasks.  While  the  co-existence  of  well-  and  ill- 
defined  tasks  in  a  single  domain  is  nearly  universally  agreed  upon  by  researchers, 
this  key  point  is  often  quickly  buried  by  an  extensive  discussion  about  what  makes 
certain  domain  tasks  ill  defined  (e.g.,  disagreement  about  ideal  solutions,  multiple 
solution  paths). 

In  this  chapter,  we  first  take  a  step  back  to  consider  what  is  meant  by  a  domain  in 
the  context  of  learning.  Next,  based  on  this  definition,  we  map  out  the  components 
that  are  in  a  learning  domain,  since  each  component  may  have  ill-defined  parts. 
This  leads  into  a  discussion  about  the  strategies  that  have  been  used  to  make  ill- 
defined  domains  tractable  for  certain  types  of  pedagogy.  Examples  of  ITS  research 
that  applies  these  strategies  are  noted.  Finally,  we  conclude  with  practical  how-to 
considerations  and  open  research  questions  for  approaching  ill-defined  domains. 
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A  Review  of  Self-Reference  and  Context  Personalization  in 
Different  Computer-Based  Educational  Domains 


Abstract:  A  goal  of  instruction  is  for  a  student  to  not  only  learn  the  material  and 
be  able  to  recall  it  immediately  afterwards,  but  also  to  be  able  to  retain  it  for  future 
use.  Additionally,  it  is  preferable  for  the  material  to  be  taught  in  such  a  way  that  the 
student  can  use  what  was  learned  and  then  apply  or  transfer  it  to  new  situations. 
Human  memory  and  learning  have  been  extensively  studied,  resulting  in 
mnemonics  and  general  strategies  that  assist  in  memory  across  different  domains 
of  instruction.  One  such  strategy  is  linking  in  formation  to  knowledge  that  the 
student  already  has  and  that  is  important  to  them.  This  strategy  can  include  linking 
information  to  the  self,  to  people  that  the  individual  knows,  or  simply  customizing 
the  materials  to  include  mention  of  topics  that  are  of  interest  to  the  individual  or 
group  of  students.  There  appears  to  be  significant  benefits  to  employing  all  of  these 
approaches;  however,  they  are  often  lumped  together  in  the  literature,  and  the 
different  domains  in  which  they  may  or  may  not  be  beneficial  have  not  been 
acknowledged. 

Many  different  domains,  topics,  and  areas  of  instruction  include  materials, 
examples,  and  practice  problems  with  wording  that  can  be  adjusted  to  make  it 
specific  to  different  students  or  interest  areas.  In  many  classes,  whether  they  are 
lecture-based,  web-based,  large  or  small,  it  is  often  difficult  to  retain  student  interest 
and  motivation  throughout.  However,  there  are  different  approaches  that  can  be 
used  to  draw  student  attention  and  personalize  the  materials  that  they  receive  in 
order  to  keep  them  more  interested  in  the  topic.  Referencing  the  self  and  context 
personalization  (changing  the  wording  of  student  materials  to  be  contextually 
consistent  with  student  interests)  has  been  shown  to  have  positive  impacts  on 
student  out  comes.  Self-reference  and  context/interest  personalization  techniques 
offer  benefits  to  memory  and  performance  without  requiring  large  adjustments  to 
learning  materials.  This  chapter  reviews  research  and  approaches  to  self-reference, 
context  personalization,  interest  personalization,  and  the  impacts  that  they  have  on 
learning  and  retention  in  different  domains.  It  has  been  assumed  that  these  strategies 
will  work  between  learning  domains;  however,  research  has  focused  primarily  on 
mathematics.  The  domains  of  instruction  are  examined  and  recommendations  are 
made  for  using  this  strategy  in  intelligent  tutoring  systems  (ITSs). 

Keywords:  Self-reference.  Context  personalization.  Computer-based  education. 
Memory  and  learning.  Interest  personalization.  Mathematics.  Intelligent  tutoring 
systems. 
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Introduction  to  Domain  Modeling  &  GIFT 

Abstract:  This  introduction  begins  with  a  description  of  tutoring  functions, 
provides  a  glimpse  of  domain  modeling  best  practices,  and  examines  the  motivation 
for  standards  in  the  design,  authoring,  instruction,  and  evaluation  of  ITS  tools  and 
methods.  We  introduce  GIFT  design  principles  and  discuss  how  readers  might  use 
this  book  as  a  design  tool.  We  begin  by  examining  the  major  components  of  TTSs 
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A  Process  for  Adaptive  Instruction  of  Tasks  in  the  Psychomotor 
Domain 

Abstract:  In  order  to  maximize  transfer  of  skills  from  the  training  environment  to 
the  work  environment,  we  advocate  closer  alignment  of  learner  experiences  in  the 
training  environment  with  those  expected  in  the  work  environment.  This  implies 
that  psychomotor  tasks  will  benefit  from  greater  transfer  if  the  training  includes 
physical  as  well  as  cognitive  elements  represented  in  the  work  environment. 
Toward  this  end,  cognitive  and  physical  measures  should  be  modeled  after 
successful  or  expert  performers  and  essential  stressors  should  be  represented  in  the 
training  environment  to  promote  realism  and  learning.  Adequate  measures  for 
assessment  and  appropriate  feedback  mechanisms  to  support  psychomotor  tasks 
should  be  represented  in  ITS  domain  models.  By  doing  this,  the  ITS  can  then  more 
effectively  align  training  and  optimize  transfer. 

Sottilare  and  LaViola  (2015)  described  an  adaptive  system  prototype  to  extend 
training  beyond  the  desktop  in  support  of  a  land  navigation  psychomotor  task  using 
smart  glasses  to  assess  progress,  augment  reality,  and  provide  feedback  to  the 
learner.  Sottilare,  Hacked,  Pike,  and  LaViola  (2016,  in  review)  described  a  similar 
adaptive  system  for  training  hemorrhage  control  tasks  using  smart  glasses  and 
pressure  sensors  to  assess  the  appropriate  application  of  tourniquets  and  pressure 
bandages.  This  chapter  expands  on  these  concepts  to  examine  interaction  design  for 
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GIFT -based  tutors  to  provide  generalized  adaptive  instruction  across  the  breadth  of 
tasks  in  the  psychomotor  domain  while  maximizing  opportunities  for  transfer. 

Keywords:  Training  environment.  Learner  experience.  Adaptive  instruction. 
Psychomotor  domain.  Intelligent  tutoring  system.  Land  navigation.  Feedback. 
Interaction  design.  Transfer  optimization. 

Citation:  Sottilare,  R.  A.,  &  LaViola,  J.  (2016).  A  Process  for  Adaptive  Instruction 
of  Tasks  in  the  Psychomotor  Domain.  Design  Recommendations  for  Intelligent 
Tutoring  Systems,  185. 

Making  Static  Lessons  Adaptive  through  Crowdsourcing  & 
Machine  Learning 

Abstract:  How  can  instructors  turn  static  text  into  adaptive  components  that  can 
be  perpetually  enhanced?  This  chapter  discusses  how  the  goals  of  instructors  and 
researchers  in  the  learning  sciences  can  be  advanced  by  integrating  crowdsourcing 
and  design  ideas  from  human-computer  interaction  (HCI)  with  machine  learning 
algorithms  that  provide  automatic  optimization.  The  approach  is  illustrated  by 
describing  a  system  that  instructors  can  use  as  a  plugin  to  improve  text  components 
of  instruction.  It  is  called  the  Adaptive  Explanation  Improvement  System  (AXIS) 
and  was  first  reported  in  Williams,  Kim,  Rafferty,  Maldonado,  Gajos,  Lasecki  & 
Heffernan  (2016a). 

AXIS  lets  an  instructor  designate  an  existing  static  explanation  to  be  turned  into  an 
adaptive,  data-driven  component  that  tests  out  different  explanations  and  improves 
over  time.  Each  component  asks  learners  to  explain  why  an  answer  to  a  problem  is 
correct,  which  promotes  reflection  and  understanding  (Williams  &  Eombrozo, 
2010),  as  is  well  established  in  education  as  the  self-explanation  effect  (Chi  et  ah, 
1989;  1994;  Eombrozo,  2006).  At  the  same  time,  learners’  explanations  can  be 
collected  and  then  presented  to  help  future  students  learn,  given  the  importance  of 
high  quality  explanations  (Renkl,  1997).  The  instructor  then  indicates  how 
explanations  should  be  compared  (e.g.,  based  on  learner  ratings  or  on  learner 
performance  on  a  related  problem),  and  AXIS  applies  a  machine  learning  algorithm 
automatically  analyze  and  use  the  data  being  collected  to  choose  better  explanations 
for  future  students.  Eor  example,  in  a  computer  science  class,  one  might  want  to 
determine  what  explanation  of  a  sorting  algorithm  is  most  clear  to  learners,  as 
measured  by  students’  rating  of  the  helpfulness  of  the  explanation.  If  learners  are 
provided  with  the  opportunity  to  rate  the  explanation  that  was  provided  to  them, 
their  responses  can  be  used  to  select  better  explanations  for  future  learners. 
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The  original  paper  (Williams  et  al.,  2016a)  provides  the  technical  details  of  the 
system  design  and  algorithm.  The  current  chapter  is  intended  to  provide  a  higher- 
level  overview  of  how  the  approach  is  relevant  to  instructors  and  researchers 
outside  of  HCI  and  machine  learning.  In  that  vein,  we  begin  with  a  brief  overview 
of  relevant  work. 

Keywords:  Static  lessons.  Adaptive  training.  Crowdsourcing.  Machine  learning. 
Data-driven.  Human-computer  interaction.  Machine  learning  algorithms. 
Automatic  optimization.  Learner  performance. 

Citation:  Williams,  J.  J.,  Kim,  J.,  Classman,  E.,  Rafferty,  A.,  &  Lasecki,  W.  S. 
(2016).  Making  Static  Lessons  Adaptive  through  Crowdsourcing  &  Machine 
Learning.  Design  Recommendations  for  Intelligent  Tutoring  Systems,  127. 

Intelligent  Tutoring  Authoring  Tools  to  Access  Training 
Effectiveness 

Abstract:  Intelligent  Tutoring  Systems  (ITS)  hold  the  potential  to  unlock  a  new  era 
of  adaptive  learner-centric  training,  but  much  of  the  current  research  focuses  on 
learner- tutor  interactions.  Alternatively,  this  paper  describes  ongoing  research  to 
support  training  effectiveness  evaluations  from  the  tutor  authoring  perspective.  The 
Army  Research  Laboratory’s  Generalized  Intelligent  Lramework  for  Tutoring 
(GILT)  is  being  extended  to  explore  training  effectiveness  within  an  ITS.  The 
resulting  prototype  operationalizes  this  framework  to  support  the  rapid,  high-level, 
visually  intuitive  analysis  of  effectiveness  at  a  user-selected  level  of  granularity  and 
to  then  offer  a  mechanism  to  delve  deeper  and  explore  individual  factors  to 
ultimately  identify  areas  for  improvement.  A  statistical  engine  is  included  to 
identify  factors  that  contribute  significantly  to  training  effectiveness  and  to  support 
experimentation  and  sensitivity  analysis. 

An  Army  marksmanship  course  was  chosen  as  a  test  case  since  it  relies  on  multiple 
training  delivery  techniques  and  includes  non-traditional  factors  such  as  trigger 
pressure  that  are  external  to  the  GILT  environment.  The  research  focuses  on  using 
the  Experience  Application  Programming  Interface  (xAPI)  formatted  data  to 
combine  disparate  data  sources,  to  support  integration  of  multiple  perspectives  and 
provide  interoperability  with  other  data  systems.  This  paper  will  describe  the 
architecture  and  integrated  tool-set  that  allows  course  authors  and  administrators  to 
more  accurately  assess  training  effectiveness  and  prioritize  targeted  improvements, 
as  well  as  sample  applications. 

Keywords:  Intelligent  Tutoring  Systems.  Data  Analytics. 
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Data  Analytics:  Techniques  and  Applications  to  Transform  Army 
Learning 

Abstract:  Data  analytics  is  the  scientific  process  of  transforming  data  into  insight 
for  making  better  decisions  and  is  used  in  industry  to  improve  organizational 
decision-making  and  in  the  sciences  to  verify  or  disprove  existing  models  or 
theories.  Current  data  analytic  models  have  begun  to  make  an  impact  on  the  way 
that  courses  are  designed,  run,  and  evaluated,  although  little  progress  has  been  made 
towards  the  design  of  a  structured  method  to  categorize  and  implement  data 
measurements  as  they  relate  to  the  Army  Learning  Model  (ALM)  goals.  The 
following  paper  describes  ongoing  work  with  the  U.S.  Army  Research  Laboratory 
to  examine  data  analytics  as  it  relates  to  the  design  of  courses,  evaluation  of 
individual  and  group  performances,  and  the  ability  to  tailor  the  learning  experience 
to  achieve  optimal  learning  outcomes.  This  paper  describes:  a)  the  methodology  for 
research  and  evaluation;  b)  the  fields  of  Learning  Analytics  and  Educational  Data 
Mining;  c)  data  analytics  methods  and  techniques  relevant  to  learning  systems;  and 
d)  a  framework  for  applying  data  analytic  methods  and  techniques  for  learning  via 
three  illustrative  use  cases.  Ultimately,  the  goal  of  this  paper  will  be  to  provide  a 
vision  for  successful  application  of  these  techniques  within  the  Army  learning 
community  and  higher  education. 

Keywords:  Data  analytics.  Army  Learning  Model  goals.  Group  performance. 
Learning  analytics.  Educational  Data  Mining. 

Citation:  Long,  R.,  Smith,  M.,  Dass,  S.,  Dillon,  C.,  &  Hill,  K.  Data  Analytics: 
Techniques  and  Applications  to  Transform  Army  Learning. 

Architecture  for  Multi-Domain  Adaptive  Training 

Abstract:  Adaptive  Training  is  intelligently  tailored,  computer-guided  experiences 
for  individuals  and  units  focused  on  optimizing  training  performance,  training 
efficiency,  deep  learning,  and  transfer  of  skills  to  the  operational  environment. 
Training  “adaptation”  is  multi-faceted.  Lor  the  trainee,  the  delivery  of  training  must 
adapt  to  individual  trainee  needs  and  to  the  organizational  groupings  of  trainees 
(e.g.,  an  Army  unit).  Training  must  be  adaptively  tailored  to  trainee  state  (cognitive, 
affective,  psychomotor,  social,  etc.)  and  to  trainee  task  performance.  Adaptations 
might  be  determined  and  delivered  in  real  time  during  training  events  or  determined 
through  assessment  of  learner  data  over  extended  time  and  delivered  periodically 
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(non-real  time).  Adaptations  may  seek  to  inform  and  optimize  instructional 
strategies  both  during  training  and  off-line  (between  training  sessions).  Training 
content  adaptations  might  be  automated,  semi-automated,  or  human  (instructor)- 
driven.  From  a  ’’training  systems”  life  cycle  perspective,  the  adaptation  approaches 
must  seek  to  optimize  training  over  learner  lifecycles  through  optimal  blending  of 
training  types  and  modalities  (e.g.,  computer-based,  tutor-based,  game -based, 
simulation-based.  Live,  Virtual,  Constructive  (LVC),  etc.). 

A  central  barrier  that  impedes  increased  use  of  adaptive  training  is  the  amount  of 
time  and  the  high  cost  required  to  build  and  maintain  these  complex  training 
applications.  This  paper  describes  an  ontology-driven  Multi-domain  Adaptive 
Training  (MAT)  approach  for  adaptive  training  application  integration  and 
management  that  targets  this  challenge.  The  MAT  approach  is  founded  on  the 
observation  that  inter-domain  ontology  mappings  may  be  used  to  drive  multi- 
domain  adaptive  training  application  information  exchange  requirements.  The 
paper  will  describe:  (i)  the  MAT  application  integration  method;  (ii)  a  conceptual 
architecture  for  providing  automation  support  for  the  MAT  method;  (iii)  an  initial 
reference  ontology  for  adaptive  training  applications  and  the  use  of  this  ontology 
for  enabling  adaptive  training  application  integration;  and  (iv)  an  illustrative  multi- 
domain  adaptive  training  application  integration  example.  The  Generalized 
Intelligent  Framework  for  Tutoring  (GIFT)  is  used  as  a  unifying  framework  for 
demonstrating  the  methods  described  in  this  paper. 

Keywords:  Adaptive  training.  Training  performance.  Training  efficiency.  Multi- 
domain  Adaptive  Training.  Generalized  Intelligent  Framework  for  Tutoring. 

Citation:  Long,  R.,  Benjamin,  P.,  Wu,  C.K.,  Fernandes,  R.,  &  Li,  B.  Architecture 
for  Multi-Domain  Adaptive  Training. 
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List  of  Symbols,  Abbreviations,  and  Acronyms 


2-D 

2-dimensional 

AAR 

after- action  review 

AAAR 

adaptive  after- action  review 

AERA 

American  Education  Research  Association 

AIS 

adaptive  instructional  system 

ALM 

Army  learning  model 

ARES 

Augmented  REality  Sandtable 

ARE 

US  Army  Research  Laboratory 

AXIS 

Adaptive  Explanation  Improvement  System 

CAT 

course  authoring  tool 

COIN 

counterinsurgency 

ECD 

evidence-centered  design 

EDA 

electrodermal  activity 

ERT 

event  reporting  tool 

GIET 

Generalized  Intelligent  Eramework  for  Tutoring 

GIETSymS 

GIET  users’  symposium 

HCI 

human-computer  interaction 

lAI 

Intelligent  Automation,  Inc 

ICM 

interoperable  competency  model 

IPA 

interpretative  phenomenological  analysis 

ISU 

Iowa  State  University 

ITS 

intelligent  tutoring  system 

KSAs 

knowledge,  skills,  and  abilities 

LEM 

learning  effect  model 

LMSs 

learning  management  systems 
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LRS 

learning  record  store 

LVC 

live,  virtual,  constructive 

MAT 

motivational  assessment  tool 

MOOCs 

massive  open  online  courses 

NCSU 

North  Carolina  State  University 

OELEs 

open-ended  learning  environments 

Recon 

reconnaissance 

RE 

reinforcement  learning 

REO 

reusable  learning  object 

ROI 

return  on  investment 

ROTC 

Army  Reserve  Officer  Training  Corps 

SAS 

survey  authoring  system 

SCRs 

skin  conductance  responses 

SEM 

structural  equation  modeling 

SME 

subject  matter  expert 

SME 

social  media  framework 

SOA 

service-  oriented  architecture 

TICE  SIG 

Technology,  Instruction,  Cognition  &  Eearning 
Special  Interest  Group 

TUI 

tutor  user  interface 

UI 

user  interfaces 

xAPI 

experience  application  program  interface 
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